From 68b8ecf68e42d63a65ef84174471a0d3889cc977 Mon Sep 17 00:00:00 2001 From: justin06lee Date: Fri, 22 May 2026 15:53:25 -0700 Subject: [PATCH 01/18] =?UTF-8?q?docs:=20spec=20for=20whiteboard=20photo?= =?UTF-8?q?=20=E2=86=92=20graph=20(Gemini-only)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Design for new vision input path: photo of a hand-drawn supply-chain graph → Gemini vision draft → editable confirm screen → grounded research enrich → existing analyzed interactive graph. Also collapses runtime to Gemini-only (drops dev/Cerebras/Tavily; keeps fake as a test-only double so CI stays offline). Co-Authored-By: Claude Opus 4.7 (1M context) --- ...26-05-22-whiteboard-photo-vision-design.md | 143 ++++++++++++++++++ 1 file changed, 143 insertions(+) create mode 100644 docs/superpowers/specs/2026-05-22-whiteboard-photo-vision-design.md diff --git a/docs/superpowers/specs/2026-05-22-whiteboard-photo-vision-design.md b/docs/superpowers/specs/2026-05-22-whiteboard-photo-vision-design.md new file mode 100644 index 0000000..584b6da --- /dev/null +++ b/docs/superpowers/specs/2026-05-22-whiteboard-photo-vision-design.md @@ -0,0 +1,143 @@ +# Whiteboard Photo → Graph (Gemini-only) + +**Date:** 2026-05-22 +**Status:** Approved, ready for implementation plan + +## Summary + +Add a new input path to zeroforce: a user draws a supply-chain graph on a physical +whiteboard, takes a photo, and uploads it. Gemini's vision reads the boxes and arrows into a +draft graph; the user reviews and corrects the detected structure on a confirm screen; then +the normal grounded extraction researches each entity to fill in weights, tiers, sectors, and +citations. The result is the same interactive analyzed graph the text input produces today. + +This work also collapses the runtime to **Gemini-only**: the `dev` (Cerebras + Tavily) run +mode is removed, and `fake` is retained strictly as an offline test double so CI and the +engine math gates keep running without GCP credentials. + +## Goals + +- Photo of a hand-drawn supply-chain graph becomes a fully analyzed, interactive graph. +- User can correct vision misreads before the expensive research pass runs. +- Everything downstream of extraction (engine, reporter, storage, history, dashboard, + simulate, what-if) is reused unchanged. +- The app runs entirely on Gemini; tests still run offline and deterministically. + +## Non-goals + +- No editable whiteboard canvas inside the app. The viz stays read-only; "whiteboard" means + the physical board the user draws on. +- No new analysis math. The RZF engine and its oracle discipline are untouched. +- No multi-image stitching or video. One photo per analysis. + +## Architecture + +Two separable parts. + +### Part A — Runtime collapse to Gemini-only + +- **`config.py` / `factory.py`:** Mode shrinks from `fake | dev | prod` to two internal + values: `prod` (default — Gemini, the only user-facing runtime) and `fake` (set only by + test fixtures). The `dev` branch is removed. +- **Providers:** Delete `cerebras.py` and `tavily.py`. Remove their dependencies from + `packages/providers/pyproject.toml`. +- **`make_search`:** Returns `FakeSearchProvider` under `fake`. In `prod`, web grounding lives + inside `GeminiProvider` (built-in Google Search), so no standalone search provider is wired. +- **`extractor/core.py`:** Delete the `dev`-only Tavily multi-query branch. Grounding for the + real runtime happens inside the Gemini provider as it does today. +- **Doc/infra sweep:** README modes table, `.env.example`, `infra/docker-compose.dev.yml`, + terraform env. `ZEROFORCE_MODE` defaults to `prod`. + +### Part B — Vision feature + +Flow: + +``` +photo + → POST /api/v1/vision (Gemini vision, synchronous, one call) + → DraftGraph (structure only) + → confirm screen (user edits nodes/edges/focal) + → POST /api/v1/analyze { seed_graph: DraftGraph } + → Gemini grounded research enriches the skeleton + → Graph + → engine + reporter + → /analyze/[id] (existing interactive dashboard) +``` + +#### Stage 1 — Vision extraction + +- **Provider protocol (`base.py`):** Add + `extract_structure_from_image(image_bytes: bytes, mime_type: str) -> DraftGraph`. +- **`GeminiProvider`:** Multimodal call — image placed in `contents` alongside a system + prompt, `response_schema=DraftGraph`. Reads boxes → nodes, arrows → directed edges, infers + the focal node. No research and no weights at this stage. +- **`FakeLLMProvider`:** Returns a deterministic canned `DraftGraph` for offline tests. +- **New schema `DraftGraph`** (in `zeroforce_schemas`): structure only. + - `nodes: list[{ id: str, label: str }]` + - `edges: list[{ source: str, target: str }]` (ids reference node ids) + - `focal_node: str` +- **New endpoint `POST /api/v1/vision`** (multipart upload): + - Validates mime type (`image/png`, `image/jpeg`, `image/webp`, `image/heic`). + - Reuses the existing 20 MB size cap. + - Runs the existing PII redaction over any text the model transcribes. + - One synchronous Gemini call; returns `DraftGraph`. Not an async job. + +#### Stage 2 — Confirm screen (web) + +- New route (e.g. `/confirm`): an editable view of the detected graph. + - Nodes: rename, add, delete. + - Edges: add, delete, reverse direction. + - Choose the focal node. + - Small live preview rendered with the existing `ForceGraph` component. +- "Analyze" button submits the confirmed `DraftGraph` to Stage 2. +- **Home page (`apps/web/app/page.tsx`):** Add a "📷 Upload whiteboard photo" affordance + beside the text box → file/camera picker → `POST /vision` → navigate to the confirm screen. + +#### Stage 3 — Enrich + analyze + +- **`AnalyzeRequest`** gains an optional field `seed_graph: DraftGraph | None`. +- When `seed_graph` is present, the extractor treats the drawn structure as the + **authoritative skeleton**. A new prompt in `prompts.py` instructs Gemini to research each + named entity and fill `weight`, `confidence`, `tier`, `sector`, and `citation`, and to add + only edges that are obviously implied — not to invent a new topology. +- When `seed_graph` is absent, behavior is exactly the text path of today. +- Everything downstream is unchanged. Output is the same `AnalysisResult`, stored and served + by `/analyze/[id]` exactly as the text flow. +- Reuses the existing async job lifecycle (202 + poll). + +## Data flow and contracts + +- `POST /api/v1/vision` (multipart `file`) → `200 DraftGraph` | `400 invalid_input` + (bad mime/size) | `502 llm_provider_error`. +- `POST /api/v1/analyze` accepts the existing body plus optional `seed_graph`. With a seed, + `input` may be a short caption/title; extraction is driven by the seed structure. +- `DraftGraph` is the single new shared contract between vision, confirm screen, and analyze. + +## Error handling + +- Unreadable photo / no graph detected: vision returns an empty or near-empty `DraftGraph`; + the confirm screen shows "nothing detected — add nodes manually or go back to text input." +- Vision provider failure: `502` with `llm_provider_error`, same shape as existing errors. +- Seed-graph extraction failure: reuses the existing corrective re-extraction retry path in + the extractor/orchestrator. + +## Testing + +- `FakeLLMProvider` gains a deterministic `extract_structure_from_image` stub. +- New tests: + - `/vision` endpoint: valid image → `DraftGraph`; oversized/bad-mime → `400`. + - Seed-graph extraction path: a `DraftGraph` skeleton drives extraction and the drawn + topology is preserved (no invented nodes). +- Engine math gates and oracle tests are untouched and continue to run offline. +- CI stays fully offline/deterministic via `fake`. + +## Defaulted micro-decisions + +- Confirm is a separate `/confirm` route, not a modal. +- `/vision` is synchronous (single fast call), not an async job. +- Internal mode names stay `prod` / `fake` (no rename of `prod` → `gemini`). + +## Out-of-scope follow-ups + +- Editable in-app whiteboard canvas. +- Multi-photo / panorama stitching for large boards. From 8bdcc83b186860334b4319f8ae17cbd733a7bc5e Mon Sep 17 00:00:00 2001 From: justin06lee Date: Fri, 22 May 2026 15:57:42 -0700 Subject: [PATCH 02/18] =?UTF-8?q?docs:=20implementation=20plan=20for=20whi?= =?UTF-8?q?teboard=20photo=20=E2=86=92=20graph?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 11 TDD tasks: DraftGraph type, Gemini-only collapse, vision provider method, seed-aware extraction, seed plumbing, /vision endpoint, web types/API, upload entry, /confirm screen, infra sweep, verification. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../2026-05-22-whiteboard-photo-vision.md | 1421 +++++++++++++++++ 1 file changed, 1421 insertions(+) create mode 100644 docs/superpowers/plans/2026-05-22-whiteboard-photo-vision.md diff --git a/docs/superpowers/plans/2026-05-22-whiteboard-photo-vision.md b/docs/superpowers/plans/2026-05-22-whiteboard-photo-vision.md new file mode 100644 index 0000000..23511e6 --- /dev/null +++ b/docs/superpowers/plans/2026-05-22-whiteboard-photo-vision.md @@ -0,0 +1,1421 @@ +# Whiteboard Photo → Graph (Gemini-only) Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Let a user photograph a hand-drawn supply-chain graph; Gemini reads it into a draft structure, the user corrects it on a confirm screen, then the normal grounded extraction enriches it into the same analyzed interactive graph the text path produces. Also collapse the runtime to Gemini-only. + +**Architecture:** A new lightweight `DraftGraph` type flows photo → `POST /api/v1/vision` (Gemini multimodal, synchronous) → confirm screen → `POST /api/v1/analyze {seed_graph}` → seed-aware extraction → existing engine/reporter/storage/dashboard. The `dev` (Cerebras/Tavily) run mode is removed; `fake` is kept only as an offline test double. + +**Tech Stack:** Python 3.12 / FastAPI / Pydantic v2 / `google-genai` (Vertex) backend; Next.js 14 (App Router) / TypeScript / Tailwind frontend; `pytest` + uv workspace. + +--- + +## File Structure + +**Created:** +- `apps/web/app/confirm/page.tsx` — editable confirm screen for the detected draft graph. +- `services/backend/tests/test_vision.py` — vision endpoint + draft-graph tests. +- `services/backend/tests/test_seed_extraction.py` — seed-graph extraction path tests. +- `packages/providers/tests/test_modes.py` — mode-collapse + provider-vision tests (create `tests/` dir if absent). + +**Modified:** +- `packages/zeroforce/zeroforce/types.py` — add `DraftNode`, `DraftEdge`, `DraftGraph`. +- `packages/providers/zeroforce_providers/base.py` — add `extract_structure_from_image` to the `LLMProvider` protocol. +- `packages/providers/zeroforce_providers/fake.py` — vision stub + seed-aware graph builder. +- `packages/providers/zeroforce_providers/gemini.py` — multimodal `extract_structure_from_image`. +- `packages/providers/zeroforce_providers/lazy.py` — forward the new method. +- `packages/providers/zeroforce_providers/factory.py` — collapse to `fake | prod`. +- `packages/providers/pyproject.toml` — drop Cerebras/Tavily deps. +- `packages/providers/zeroforce_providers/cerebras.py`, `.../tavily.py` — delete. +- `services/backend/zeroforce_backend/config.py` — default mode `prod`. +- `services/backend/zeroforce_backend/prompts.py` — seed-aware extraction prompt. +- `services/backend/zeroforce_backend/extractor/core.py` — drop `dev` branch; accept `seed`. +- `services/backend/zeroforce_backend/service_client.py` — thread `seed` through `extract`. +- `services/backend/zeroforce_backend/worker_apps.py` — thread `seed` through `/extract`. +- `services/backend/zeroforce_backend/orchestrator/core.py` — pass `req.seed_graph`; hash it into the job id. +- `services/backend/zeroforce_backend/gateway/app.py` — `POST /api/v1/vision`. +- `packages/schemas/zeroforce_schemas/models.py` — `AnalyzeRequest.seed_graph`. +- `apps/web/lib/types.ts` — `DraftGraph` interfaces. +- `apps/web/lib/api.ts` — `uploadWhiteboard`, `startAnalysis(seed)`. +- `apps/web/app/page.tsx` — photo-upload affordance. +- `README.md`, `.env.example`, `infra/docker-compose.dev.yml`, `infra/terraform/*` — Gemini-only sweep. + +--- + +## Locked interfaces (use these exact names everywhere) + +```python +# packages/zeroforce/zeroforce/types.py +class DraftNode(BaseModel): + id: str + label: str + +class DraftEdge(BaseModel): + source: str # references a DraftNode.id + target: str # references a DraftNode.id + +class DraftGraph(BaseModel): + nodes: list[DraftNode] + edges: list[DraftEdge] + focal_node: str +``` + +```python +# provider method (LLMProvider protocol) +async def extract_structure_from_image(self, image_bytes: bytes, mime_type: str) -> DraftGraph: ... +``` + +``` +# prompt seed marker (exact literal, parsed by the fake provider) +SEED_STRUCTURE_JSON: +``` + +--- + +## Task 1: `DraftGraph` core type + +**Files:** +- Modify: `packages/zeroforce/zeroforce/types.py` +- Test: `packages/zeroforce/tests/test_properties.py` (append one test) — or create `packages/zeroforce/tests/test_draftgraph.py` + +- [ ] **Step 1: Write the failing test** + +Create `packages/zeroforce/tests/test_draftgraph.py`: + +```python +from zeroforce.types import DraftEdge, DraftGraph, DraftNode + + +def test_draftgraph_roundtrips(): + dg = DraftGraph( + nodes=[DraftNode(id="apple", label="Apple"), DraftNode(id="tsmc", label="TSMC")], + edges=[DraftEdge(source="tsmc", target="apple")], + focal_node="apple", + ) + again = DraftGraph.model_validate_json(dg.model_dump_json()) + assert again.focal_node == "apple" + assert [n.id for n in again.nodes] == ["apple", "tsmc"] + assert again.edges[0].source == "tsmc" +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd packages/zeroforce && uv run pytest tests/test_draftgraph.py -v` +Expected: FAIL with `ImportError: cannot import name 'DraftGraph'`. + +- [ ] **Step 3: Add the types** + +In `packages/zeroforce/zeroforce/types.py`, after the `Edge` class and before `Graph`, add: + +```python +class DraftNode(BaseModel): + """A node detected from a hand-drawn whiteboard graph (structure only, pre-research).""" + + id: str + label: str + + +class DraftEdge(BaseModel): + """A directed arrow detected from a whiteboard graph; source/target reference DraftNode.id.""" + + source: str + target: str + + +class DraftGraph(BaseModel): + """Vision output: the boxes + arrows read off a whiteboard, before research enrichment.""" + + nodes: list[DraftNode] + edges: list[DraftEdge] + focal_node: str +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `cd packages/zeroforce && uv run pytest tests/test_draftgraph.py -v` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add packages/zeroforce/zeroforce/types.py packages/zeroforce/tests/test_draftgraph.py +git commit -m "feat(types): add DraftGraph for whiteboard vision structure" +``` + +--- + +## Task 2: Collapse runtime to Gemini-only (`fake | prod`) + +**Files:** +- Modify: `packages/providers/zeroforce_providers/factory.py` +- Modify: `services/backend/zeroforce_backend/config.py:42` +- Modify: `services/backend/zeroforce_backend/extractor/core.py` +- Delete: `packages/providers/zeroforce_providers/cerebras.py`, `packages/providers/zeroforce_providers/tavily.py` +- Modify: `packages/providers/pyproject.toml` +- Test: `packages/providers/tests/test_modes.py` + +- [ ] **Step 1: Write the failing test** + +Create `packages/providers/tests/test_modes.py`: + +```python +import pytest + +from zeroforce_providers.factory import get_mode, make_llm, make_search +from zeroforce_providers.fake import FakeLLMProvider, FakeSearchProvider + + +def test_dev_mode_rejected(monkeypatch): + monkeypatch.setenv("ZEROFORCE_MODE", "dev") + with pytest.raises(ValueError): + get_mode() + + +def test_default_mode_is_prod(monkeypatch): + monkeypatch.delenv("ZEROFORCE_MODE", raising=False) + assert get_mode() == "prod" + + +def test_fake_mode_returns_fakes(): + assert isinstance(make_llm("fake"), FakeLLMProvider) + assert isinstance(make_search("fake"), FakeSearchProvider) +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd packages/providers && uv run pytest tests/test_modes.py -v` +Expected: FAIL — `test_dev_mode_rejected` fails (dev still accepted) and/or `test_default_mode_is_prod` fails (default is `fake`). + +- [ ] **Step 3: Rewrite the factory** + +Replace the entire contents of `packages/providers/zeroforce_providers/factory.py` with: + +```python +"""Provider factory — Gemini-only runtime with a fake test double. + +Mode is a single env var `ZEROFORCE_MODE`: + prod (default) -> Gemini (Vertex, GCP ADC); web grounding is built into the provider. + fake -> deterministic offline providers; set only by tests/CI. No keys, no network. +""" + +from __future__ import annotations + +import os +from typing import Literal + +from .base import LLMProvider, SearchProvider + +Mode = Literal["fake", "prod"] + + +def get_mode() -> Mode: + mode = os.environ.get("ZEROFORCE_MODE", "prod").lower() + if mode not in ("fake", "prod"): + raise ValueError(f"ZEROFORCE_MODE must be fake|prod, got {mode!r}") + return mode # type: ignore[return-value] + + +def make_llm(mode: Mode | None = None) -> LLMProvider: + mode = mode or get_mode() + if mode == "fake": + from .fake import FakeLLMProvider + + return FakeLLMProvider() + from .gemini import GeminiProvider + + return GeminiProvider() + + +def make_search(mode: Mode | None = None) -> SearchProvider: + """Only the fake double needs a standalone search provider; in prod, web grounding lives + inside the Gemini provider (built-in Google Search), so this returns a fake placeholder + that the prod extraction path never calls.""" + mode = mode or get_mode() + from .fake import FakeSearchProvider + + return FakeSearchProvider() +``` + +- [ ] **Step 4: Default the backend mode to `prod`** + +In `services/backend/zeroforce_backend/config.py`, change line 42: + +```python + mode=os.environ.get("ZEROFORCE_MODE", "fake").lower(), +``` + +to: + +```python + mode=os.environ.get("ZEROFORCE_MODE", "prod").lower(), +``` + +Also update the `Settings.mode` comment on line 27 from `# fake | dev | prod` to `# prod (Gemini) | fake (tests only)`. + +- [ ] **Step 5: Remove the `dev` branch from the extractor** + +Replace the entire contents of `services/backend/zeroforce_backend/extractor/core.py` with: + +```python +"""Extractor service core — graph extraction with grounding (spec §3.3 step 4, §8.1). + +- prod: one Gemini call whose two-pass grounding (Google Search) happens inside the provider. +- fake: one structured call (the fake provider builds the graph deterministically). + +When `seed` is provided (a whiteboard DraftGraph), it is embedded in the prompt as the +authoritative skeleton and the model researches the named entities to fill weights, tiers, +sectors, and citations rather than inventing a fresh topology. +""" + +from __future__ import annotations + +from zeroforce.types import DraftGraph, Graph +from zeroforce_providers import LLMProvider, SearchProvider + +from ..prompts import EXTRACTION_SYSTEM, extraction_user + + +async def extract_graph( + user_input: str, + *, + mode: str, + max_nodes: int, + llm: LLMProvider, + search: SearchProvider, + correction: str = "", + seed: DraftGraph | None = None, +) -> Graph: + user = extraction_user( + user_input, max_nodes=max_nodes, grounding="", correction=correction, seed=seed + ) + graph = await llm.generate_structured(EXTRACTION_SYSTEM, user, Graph, temperature=0.2) + return graph +``` + +(Note: `extraction_user` gains a `seed` parameter in Task 4; this file imports it now and the signature is added there. If executing strictly in order, the `seed=seed` kwarg will raise `TypeError` until Task 4 — that is expected and Task 4's tests cover it. To keep each task green in isolation, do Task 4 immediately after this step before running the full suite.) + +- [ ] **Step 6: Delete the dead providers** + +```bash +git rm packages/providers/zeroforce_providers/cerebras.py packages/providers/zeroforce_providers/tavily.py +``` + +- [ ] **Step 7: Drop their dependencies** + +In `packages/providers/pyproject.toml`, remove any `cerebras`/`cerebras-cloud-sdk` and `tavily`/`tavily-python` entries from `dependencies` / optional-dependency groups. Run: + +Run: `cd packages/providers && grep -in "cerebras\|tavily" pyproject.toml` +Expected: no matches after editing. + +- [ ] **Step 8: Run mode tests** + +Run: `cd packages/providers && uv run pytest tests/test_modes.py -v` +Expected: PASS (all three). + +- [ ] **Step 9: Commit** + +```bash +git add -A packages/providers services/backend/zeroforce_backend/config.py services/backend/zeroforce_backend/extractor/core.py +git commit -m "feat(runtime): collapse to Gemini-only; keep fake as test double" +``` + +--- + +## Task 3: Provider vision method (`extract_structure_from_image`) + +**Files:** +- Modify: `packages/providers/zeroforce_providers/base.py` +- Modify: `packages/providers/zeroforce_providers/fake.py` +- Modify: `packages/providers/zeroforce_providers/gemini.py` +- Modify: `packages/providers/zeroforce_providers/lazy.py` +- Test: `packages/providers/tests/test_modes.py` (append) + +- [ ] **Step 1: Write the failing test** + +Append to `packages/providers/tests/test_modes.py`: + +```python +import asyncio + +from zeroforce.types import DraftGraph +from zeroforce_providers.lazy import LazyLLM + + +def test_fake_vision_is_deterministic(): + llm = make_llm("fake") + png = b"\x89PNG\r\n\x1a\n" + b"fake-image-bytes" + a = asyncio.run(llm.extract_structure_from_image(png, "image/png")) + b = asyncio.run(llm.extract_structure_from_image(png, "image/png")) + assert isinstance(a, DraftGraph) + assert a.model_dump() == b.model_dump() + assert a.focal_node in {n.id for n in a.nodes} + assert len(a.nodes) >= 2 + + +def test_lazy_forwards_vision(): + llm = LazyLLM(lambda: make_llm("fake"), label="fake") + dg = asyncio.run(llm.extract_structure_from_image(b"x", "image/png")) + assert isinstance(dg, DraftGraph) +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd packages/providers && uv run pytest tests/test_modes.py::test_fake_vision_is_deterministic -v` +Expected: FAIL — `AttributeError: 'FakeLLMProvider' object has no attribute 'extract_structure_from_image'`. + +- [ ] **Step 3: Add the protocol method** + +In `packages/providers/zeroforce_providers/base.py`, add an import and a method. Change the import block near the top to also import `DraftGraph`: + +```python +from zeroforce.types import DraftGraph +``` + +Inside the `LLMProvider` Protocol (after `generate_text`, before `health`), add: + +```python + async def extract_structure_from_image( + self, + image_bytes: bytes, + mime_type: str, + ) -> DraftGraph: ... +``` + +- [ ] **Step 4: Implement the fake vision stub** + +In `packages/providers/zeroforce_providers/fake.py`, update the import line `from zeroforce.types import Edge, Graph, Node` to: + +```python +from zeroforce.types import DraftEdge, DraftGraph, DraftNode, Edge, Graph, Node +``` + +Add a module-level helper after `build_fake_graph` (before `class FakeLLMProvider`): + +```python +def build_fake_draft(image_bytes: bytes) -> DraftGraph: + """Deterministic whiteboard-vision stub: a tiny fixed 3-node draft, independent of the + bytes' content, so vision tests are byte-stable offline.""" + nodes = [ + DraftNode(id="focal", label="Focal Co"), + DraftNode(id="supplier_a", label="Supplier A"), + DraftNode(id="material_x", label="Material X"), + ] + edges = [ + DraftEdge(source="supplier_a", target="focal"), + DraftEdge(source="material_x", target="supplier_a"), + ] + return DraftGraph(nodes=nodes, edges=edges, focal_node="focal") +``` + +Then add this method to `FakeLLMProvider` (after `generate_text`): + +```python + async def extract_structure_from_image(self, image_bytes, mime_type) -> DraftGraph: + return build_fake_draft(image_bytes) +``` + +- [ ] **Step 5: Implement the Gemini vision method** + +In `packages/providers/zeroforce_providers/gemini.py`, add the import at the top (with the other `from __future__`-adjacent imports is fine; keep lazy SDK imports inside the method): + +```python +from zeroforce.types import DraftGraph +``` + +Add this method to `GeminiProvider` (after `generate_text`, before `health`): + +```python + async def extract_structure_from_image(self, image_bytes, mime_type) -> DraftGraph: + """Read a hand-drawn supply-chain graph off a photo into a DraftGraph (structure only). + + Single multimodal call: image + instruction, response constrained to DraftGraph. No web + research here — that happens later in the seed-aware extraction pass. + """ + from google.genai.types import Part + + system = ( + "You read hand-drawn supply-chain diagrams from photos. Each box/label is a node; " + "each arrow is a directed edge pointing from supplier to customer. Transcribe the " + "drawing literally into the DraftGraph schema. Use lowercase slug ids derived from " + "labels (e.g. 'TSMC' -> 'tsmc'). Pick the most-downstream box (the one most arrows " + "point toward, or labelled as the focal firm/product) as focal_node. Do not invent " + "nodes or edges that are not drawn." + ) + image_part = Part.from_bytes(data=image_bytes, mime_type=mime_type) + cfg = self._cfg( + system_instruction=system, + temperature=0.0, + response_mime_type="application/json", + response_schema=DraftGraph, + ) + resp = await self.client.aio.models.generate_content( + model=self.model, + contents=[image_part, "Transcribe this whiteboard supply-chain drawing."], + config=cfg, + ) + return DraftGraph.model_validate_json(resp.text) +``` + +- [ ] **Step 6: Forward through LazyLLM** + +In `packages/providers/zeroforce_providers/lazy.py`, add to `LazyLLM` (after `generate_text`): + +```python + async def extract_structure_from_image(self, *a, **k): + return await self._get().extract_structure_from_image(*a, **k) +``` + +- [ ] **Step 7: Run vision tests** + +Run: `cd packages/providers && uv run pytest tests/test_modes.py -v` +Expected: PASS (all five). The Gemini implementation is not exercised offline; it is verified manually against Vertex in Task 11's notes. + +- [ ] **Step 8: Commit** + +```bash +git add packages/providers/zeroforce_providers/base.py packages/providers/zeroforce_providers/fake.py packages/providers/zeroforce_providers/gemini.py packages/providers/zeroforce_providers/lazy.py packages/providers/tests/test_modes.py +git commit -m "feat(providers): add extract_structure_from_image vision method" +``` + +--- + +## Task 4: Seed-aware extraction prompt + fake builder + +**Files:** +- Modify: `services/backend/zeroforce_backend/prompts.py:14` +- Modify: `packages/providers/zeroforce_providers/fake.py` +- Test: `services/backend/tests/test_seed_extraction.py` + +- [ ] **Step 1: Write the failing test** + +Create `services/backend/tests/test_seed_extraction.py`: + +```python +import asyncio + +from zeroforce.types import DraftEdge, DraftGraph, DraftNode +from zeroforce_backend.prompts import extraction_user +from zeroforce_providers.fake import FakeLLMProvider +from zeroforce.types import Graph + + +def _seed(): + return DraftGraph( + nodes=[DraftNode(id="acme", label="Acme"), DraftNode(id="widgetco", label="WidgetCo")], + edges=[DraftEdge(source="widgetco", target="acme")], + focal_node="acme", + ) + + +def test_prompt_embeds_seed_marker(): + prompt = extraction_user("Acme", max_nodes=12, seed=_seed()) + assert "SEED_STRUCTURE_JSON:" in prompt + assert "widgetco" in prompt + + +def test_fake_preserves_seed_topology(): + prompt = extraction_user("Acme", max_nodes=12, seed=_seed()) + graph = asyncio.run(FakeLLMProvider().generate_structured("sys", prompt, Graph)) + assert graph.focal_node == "acme" + assert {n.id for n in graph.nodes} == {"acme", "widgetco"} + assert any(e.source == "widgetco" and e.target == "acme" for e in graph.edges) + + +def test_fake_without_seed_unchanged(): + prompt = extraction_user("Apple semiconductor", max_nodes=12) + graph = asyncio.run(FakeLLMProvider().generate_structured("sys", prompt, Graph)) + # No seed marker -> the normal deterministic multi-tier fake graph (tier-0 focal present). + assert any(n.tier == 0 for n in graph.nodes) +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd services/backend && uv run pytest tests/test_seed_extraction.py -v` +Expected: FAIL — `extraction_user()` has no `seed` keyword (`TypeError`). + +- [ ] **Step 3: Add the `seed` parameter to the prompt** + +In `services/backend/zeroforce_backend/prompts.py`, add the import at the top (after the module docstring): + +```python +from zeroforce.types import DraftGraph +``` + +Replace the `extraction_user` signature and body opening (lines 14-18) so it accepts `seed` and emits a seed block. Change: + +```python +def extraction_user(user_input: str, max_nodes: int, grounding: str = "", correction: str = "") -> str: + grounding_block = f"\n\nResearched findings to ground the graph in real, named companies:\n{grounding}\n" if grounding else "" + correction_block = f"\n\nCORRECTION — your previous attempt was rejected:\n{correction}\n" if correction else "" + return f"""\ +Build a MULTI-TIER supply chain dependency graph for: {user_input}{grounding_block}{correction_block} +``` + +to: + +```python +def extraction_user( + user_input: str, + max_nodes: int, + grounding: str = "", + correction: str = "", + seed: "DraftGraph | None" = None, +) -> str: + grounding_block = f"\n\nResearched findings to ground the graph in real, named companies:\n{grounding}\n" if grounding else "" + correction_block = f"\n\nCORRECTION — your previous attempt was rejected:\n{correction}\n" if correction else "" + seed_block = "" + if seed is not None: + seed_block = ( + "\n\nThe user drew this supply-chain structure on a whiteboard. Treat it as the " + "AUTHORITATIVE SKELETON: keep these exact nodes and the drawn arrow directions, " + "research each named entity to fill weights/confidence/tier/sector/citations, and " + "add only edges that are obviously implied. Do NOT invent a different topology.\n" + f"SEED_STRUCTURE_JSON:\n{seed.model_dump_json()}\n" + ) + return f"""\ +Build a MULTI-TIER supply chain dependency graph for: {user_input}{grounding_block}{correction_block}{seed_block} +``` + +(The rest of the f-string — "This must be a realistic…" through "Return only the structured Graph schema." — stays exactly as is.) + +- [ ] **Step 4: Make the fake provider seed-aware** + +In `packages/providers/zeroforce_providers/fake.py`, add a builder after `build_fake_draft`: + +```python +def build_fake_graph_from_seed(seed: DraftGraph) -> Graph: + """Turn a whiteboard DraftGraph into a full Graph deterministically, preserving the drawn + topology. Tiers are BFS distance (in reversed edges) from the focal sink; incoming weights + per target are normalized to sum to 1.0 (first edge absorbs the remainder).""" + ids = [n.id for n in seed.nodes] + label = {n.id: n.label for n in seed.nodes} + + # tier = shortest distance from focal walking edges backwards (supplier side deeper). + incoming: dict[str, list[str]] = {i: [] for i in ids} + for e in seed.edges: + if e.target in incoming and e.source in incoming: + incoming[e.target].append(e.source) + tier: dict[str, int] = {seed.focal_node: 0} + frontier = [seed.focal_node] + while frontier: + nxt: list[str] = [] + for node in frontier: + for supplier in incoming.get(node, []): + if supplier not in tier: + tier[supplier] = tier[node] + 1 + nxt.append(supplier) + frontier = nxt + for i in ids: + tier.setdefault(i, 1) + + sector_for = {0: "focal", 1: "assembly", 2: "components", 3: "materials"} + nodes = [ + Node(id=i, label=label[i], sector=sector_for.get(min(tier[i], 3), "materials"), tier=tier[i]) + for i in ids + ] + + by_target: dict[str, list[str]] = {} + for e in seed.edges: + if e.target in incoming and e.source in incoming: + by_target.setdefault(e.target, []).append(e.source) + + edges: list[Edge] = [] + for dst, sources in by_target.items(): + n = len(sources) + base = 1.0 / n + for idx, src in enumerate(sources): + w = (1.0 - base * (n - 1)) if idx == 0 else base + edges.append(Edge( + source=src, target=dst, weight=w, confidence="medium", + citation="whiteboard (fake seed)", + reasoning=f"{label.get(src, src)} supplies {label.get(dst, dst)} (drawn).", + )) + + return Graph( + nodes=nodes, edges=edges, focal_node=seed.focal_node, + extraction_model="fake/seed-v1", extraction_timestamp=_FIXED_TS, + ) +``` + +Then update `FakeLLMProvider.generate_structured` to honor an embedded seed. Replace: + +```python + async def generate_structured(self, system, user, schema, temperature=0.2, **_): + if schema is Graph: + return build_fake_graph(user, max_nodes=self._max_nodes) + # Generic fallback: instantiate the schema from a deterministic empty-ish payload. + return _instantiate_minimal(schema) +``` + +with: + +```python + async def generate_structured(self, system, user, schema, temperature=0.2, **_): + if schema is Graph: + marker = "SEED_STRUCTURE_JSON:" + if marker in user: + blob = user.split(marker, 1)[1].strip().splitlines()[0] + return build_fake_graph_from_seed(DraftGraph.model_validate_json(blob)) + return build_fake_graph(user, max_nodes=self._max_nodes) + # Generic fallback: instantiate the schema from a deterministic empty-ish payload. + return _instantiate_minimal(schema) +``` + +- [ ] **Step 5: Run seed tests** + +Run: `cd services/backend && uv run pytest tests/test_seed_extraction.py -v` +Expected: PASS (all four). Also re-run the extractor import smoke from Task 2: +Run: `cd services/backend && uv run python -c "from zeroforce_backend.extractor.core import extract_graph"` +Expected: no error. + +- [ ] **Step 6: Commit** + +```bash +git add services/backend/zeroforce_backend/prompts.py packages/providers/zeroforce_providers/fake.py services/backend/tests/test_seed_extraction.py +git commit -m "feat(extractor): seed-aware extraction from whiteboard DraftGraph" +``` + +--- + +## Task 5: Thread `seed_graph` through request → orchestrator → clients + +**Files:** +- Modify: `packages/schemas/zeroforce_schemas/models.py:26-33` +- Modify: `services/backend/zeroforce_backend/service_client.py` +- Modify: `services/backend/zeroforce_backend/worker_apps.py:32-38` +- Modify: `services/backend/zeroforce_backend/orchestrator/core.py` +- Test: `services/backend/tests/test_seed_extraction.py` (append) + +- [ ] **Step 1: Write the failing test** + +Append to `services/backend/tests/test_seed_extraction.py`: + +```python +def test_analyze_request_accepts_seed(): + from zeroforce_schemas import AnalyzeRequest + + req = AnalyzeRequest(input="Acme", seed_graph=_seed()) + assert req.seed_graph is not None + assert req.seed_graph.focal_node == "acme" + + req2 = AnalyzeRequest(input="Acme") + assert req2.seed_graph is None + + +def test_inprocess_client_passes_seed(): + from zeroforce_backend.service_client import InProcessServiceClient + from zeroforce_providers.fake import FakeLLMProvider, FakeSearchProvider + + client = InProcessServiceClient("fake", FakeLLMProvider(), FakeSearchProvider()) + graph = asyncio.run(client.extract("Acme", 12, "", _seed())) + assert {n.id for n in graph.nodes} == {"acme", "widgetco"} +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd services/backend && uv run pytest tests/test_seed_extraction.py::test_analyze_request_accepts_seed tests/test_seed_extraction.py::test_inprocess_client_passes_seed -v` +Expected: FAIL — `AnalyzeRequest` has no `seed_graph`; `InProcessServiceClient.extract` takes no 4th positional arg. + +- [ ] **Step 3: Add `seed_graph` to `AnalyzeRequest`** + +In `packages/schemas/zeroforce_schemas/models.py`, update the import on line 11: + +```python +from zeroforce.types import Graph, NodeAnalysis +``` + +to: + +```python +from zeroforce.types import DraftGraph, Graph, NodeAnalysis +``` + +Add a field to `AnalyzeRequest` (after `wait` on line 33): + +```python + seed_graph: DraftGraph | None = None # whiteboard-drawn skeleton; drives extraction when set +``` + +- [ ] **Step 4: Thread `seed` through the ServiceClient protocol + both impls** + +In `services/backend/zeroforce_backend/service_client.py`: + +Update the import on line 14: + +```python +from zeroforce.types import Graph, NodeAnalysis +``` + +to: + +```python +from zeroforce.types import DraftGraph, Graph, NodeAnalysis +``` + +Change the protocol method (line 24): + +```python + async def extract(self, user_input: str, max_nodes: int, correction: str = "") -> Graph: ... +``` + +to: + +```python + async def extract(self, user_input: str, max_nodes: int, correction: str = "", seed: DraftGraph | None = None) -> Graph: ... +``` + +Change `InProcessServiceClient.extract` (lines 36-40): + +```python + async def extract(self, user_input: str, max_nodes: int, correction: str = "") -> Graph: + return await extract_graph( + user_input, mode=self.mode, max_nodes=max_nodes, llm=self.llm, search=self.search, + correction=correction, + ) +``` + +to: + +```python + async def extract(self, user_input: str, max_nodes: int, correction: str = "", seed: DraftGraph | None = None) -> Graph: + return await extract_graph( + user_input, mode=self.mode, max_nodes=max_nodes, llm=self.llm, search=self.search, + correction=correction, seed=seed, + ) +``` + +Change `HTTPServiceClient.extract` (lines 70-75): + +```python + async def extract(self, user_input: str, max_nodes: int, correction: str = "") -> Graph: + data = await self._post( + "extractor", "/extract", + {"input": user_input, "max_nodes": max_nodes, "correction": correction}, + ) + return Graph.model_validate(data) +``` + +to: + +```python + async def extract(self, user_input: str, max_nodes: int, correction: str = "", seed: DraftGraph | None = None) -> Graph: + data = await self._post( + "extractor", "/extract", + {"input": user_input, "max_nodes": max_nodes, "correction": correction, + "seed": seed.model_dump(mode="json") if seed is not None else None}, + ) + return Graph.model_validate(data) +``` + +- [ ] **Step 5: Accept `seed` in the extractor worker app** + +In `services/backend/zeroforce_backend/worker_apps.py`, update the import on line 12: + +```python +from zeroforce.types import Graph, NodeAnalysis +``` + +to: + +```python +from zeroforce.types import DraftGraph, Graph, NodeAnalysis +``` + +Replace the `/extract` handler (lines 32-38): + +```python + @app.post("/extract") + async def extract(payload: dict): + graph = await extract_graph( + payload["input"], mode=settings.mode, max_nodes=payload.get("max_nodes", 12), + llm=llm, search=search, correction=payload.get("correction", ""), + ) + return graph.model_dump(mode="json") +``` + +with: + +```python + @app.post("/extract") + async def extract(payload: dict): + seed_raw = payload.get("seed") + seed = DraftGraph.model_validate(seed_raw) if seed_raw else None + graph = await extract_graph( + payload["input"], mode=settings.mode, max_nodes=payload.get("max_nodes", 12), + llm=llm, search=search, correction=payload.get("correction", ""), seed=seed, + ) + return graph.model_dump(mode="json") +``` + +- [ ] **Step 6: Pass the seed in the orchestrator + hash it into the job id** + +In `services/backend/zeroforce_backend/orchestrator/core.py`: + +Update `_job_id` (lines 55-65) to include the seed so different drawings don't collide: + +```python + def _job_id(self, req: AnalyzeRequest) -> str: + return content_hash( + { + "input": req.input, + "mode": req.mode, + "max_nodes": req.max_nodes, + "confidence_threshold": req.confidence_threshold, + "enable_validation_pass": req.enable_validation_pass, + "engine": self.settings.mode, + "seed": req.seed_graph.model_dump(mode="json") if req.seed_graph else None, + } + ) +``` + +Update the extract call inside `_run_pipeline` (line 138): + +```python + graph = await self.client.extract(req.input, req.max_nodes, correction) +``` + +to: + +```python + graph = await self.client.extract(req.input, req.max_nodes, correction, req.seed_graph) +``` + +- [ ] **Step 7: Run the seed tests** + +Run: `cd services/backend && uv run pytest tests/test_seed_extraction.py -v` +Expected: PASS (all six). + +- [ ] **Step 8: Commit** + +```bash +git add packages/schemas/zeroforce_schemas/models.py services/backend/zeroforce_backend/service_client.py services/backend/zeroforce_backend/worker_apps.py services/backend/zeroforce_backend/orchestrator/core.py services/backend/tests/test_seed_extraction.py +git commit -m "feat(pipeline): thread whiteboard seed_graph through analyze" +``` + +--- + +## Task 6: `POST /api/v1/vision` endpoint + +**Files:** +- Modify: `services/backend/zeroforce_backend/gateway/app.py` +- Test: `services/backend/tests/test_vision.py` + +- [ ] **Step 1: Write the failing test** + +Create `services/backend/tests/test_vision.py`: + +```python +import io + +from fastapi.testclient import TestClient + +from zeroforce_backend.config import Settings +from zeroforce_backend.gateway.app import build_context, create_app + + +def _client(): + ctx = build_context(Settings(mode="fake", db_path=":memory:")) + return TestClient(create_app(ctx)) + + +def test_vision_returns_draftgraph(): + client = _client() + png = b"\x89PNG\r\n\x1a\n" + b"drawn-graph" + resp = client.post( + "/api/v1/vision", + files={"file": ("board.png", io.BytesIO(png), "image/png")}, + ) + assert resp.status_code == 200 + body = resp.json() + assert "nodes" in body and "edges" in body and "focal_node" in body + assert body["focal_node"] in {n["id"] for n in body["nodes"]} + + +def test_vision_rejects_non_image(): + client = _client() + resp = client.post( + "/api/v1/vision", + files={"file": ("notes.txt", io.BytesIO(b"hello"), "text/plain")}, + ) + assert resp.status_code == 400 + assert resp.json()["detail"]["code"] == "invalid_input" +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd services/backend && uv run pytest tests/test_vision.py -v` +Expected: FAIL — `404` (no `/api/v1/vision` route). + +- [ ] **Step 3: Add the endpoint** + +In `services/backend/zeroforce_backend/gateway/app.py`, add the constant after `_PII` (line 40): + +```python +_IMAGE_MIMES = {"image/png", "image/jpeg", "image/webp", "image/heic", "image/heif"} +``` + +Add the route immediately after the `upload` handler (after line 195, before `return app`): + +```python + @app.post("/api/v1/vision") + async def vision(file: UploadFile, _: None = Depends(require_auth)): + mime = (file.content_type or "").lower() + if mime not in _IMAGE_MIMES: + raise HTTPException(status_code=400, detail={"code": "invalid_input", + "message": f"unsupported image type {mime!r}"}) + raw = await file.read() + if len(raw) > 20 * 1024 * 1024: + raise HTTPException(status_code=400, detail={"code": "invalid_input", "message": "file > 20MB"}) + try: + draft = await ctx.llm.extract_structure_from_image(raw, mime) + except Exception as e: # noqa: BLE001 - provider/parse failure -> clean 502 + raise HTTPException(status_code=502, detail={"code": "llm_provider_error", + "message": f"{type(e).__name__}: {e}"}) + return json.loads(draft.model_dump_json()) +``` + +- [ ] **Step 4: Run the vision tests** + +Run: `cd services/backend && uv run pytest tests/test_vision.py -v` +Expected: PASS (both). + +- [ ] **Step 5: Commit** + +```bash +git add services/backend/zeroforce_backend/gateway/app.py services/backend/tests/test_vision.py +git commit -m "feat(api): POST /api/v1/vision — photo to DraftGraph" +``` + +--- + +## Task 7: Web — DraftGraph types + API helpers + +**Files:** +- Modify: `apps/web/lib/types.ts` +- Modify: `apps/web/lib/api.ts` + +- [ ] **Step 1: Add the draft types** + +Append to `apps/web/lib/types.ts`: + +```typescript +export interface DraftNode { + id: string; + label: string; +} + +export interface DraftEdge { + source: string; + target: string; +} + +export interface DraftGraph { + nodes: DraftNode[]; + edges: DraftEdge[]; + focal_node: string; +} +``` + +- [ ] **Step 2: Add the API helpers** + +In `apps/web/lib/api.ts`, update the import on line 1: + +```typescript +import type { AnalysisSummary, JobRecord, SimulateResponse } from "./types"; +``` + +to: + +```typescript +import type { AnalysisSummary, DraftGraph, JobRecord, SimulateResponse } from "./types"; +``` + +Replace `startAnalysis` (lines 12-19): + +```typescript +export async function startAnalysis(input: string, mode = "company"): Promise<{ id: string }> { + const res = await fetch("/api/v1/analyze", { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ input, mode, max_nodes: 16 }), + }); + return jsonOrThrow(res); +} +``` + +with: + +```typescript +export async function startAnalysis( + input: string, + mode = "company", + seedGraph?: DraftGraph, +): Promise<{ id: string }> { + const res = await fetch("/api/v1/analyze", { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ input, mode, max_nodes: 16, seed_graph: seedGraph ?? null }), + }); + return jsonOrThrow(res); +} + +export async function uploadWhiteboard(file: File): Promise { + const form = new FormData(); + form.append("file", file); + const res = await fetch("/api/v1/vision", { method: "POST", body: form }); + return jsonOrThrow(res) as Promise; +} +``` + +- [ ] **Step 3: Type-check** + +Run: `cd apps/web && npx tsc --noEmit` +Expected: no errors. + +- [ ] **Step 4: Commit** + +```bash +git add apps/web/lib/types.ts apps/web/lib/api.ts +git commit -m "feat(web): DraftGraph types + vision/seed API helpers" +``` + +--- + +## Task 8: Web — photo-upload affordance on the home page + +**Files:** +- Modify: `apps/web/app/page.tsx` + +- [ ] **Step 1: Add upload state + handler** + +In `apps/web/app/page.tsx`, update the import on line 6: + +```typescript +import { startAnalysis } from "@/lib/api"; +``` + +to: + +```typescript +import { startAnalysis, uploadWhiteboard } from "@/lib/api"; +``` + +Inside `Home`, after the existing `submit` function (after line 31), add: + +```typescript + async function onPhoto(e: React.ChangeEvent) { + const file = e.target.files?.[0]; + if (!file) return; + setLoading(true); + setError(null); + try { + const draft = await uploadWhiteboard(file); + sessionStorage.setItem("zeroforce.draft", JSON.stringify(draft)); + router.push("/confirm"); + } catch (err) { + setError((err as Error).message); + setLoading(false); + } + } +``` + +- [ ] **Step 2: Add the upload control to the form** + +In `apps/web/app/page.tsx`, immediately after the closing `` (after line 67), add: + +```tsx +
+ +
+``` + +- [ ] **Step 3: Type-check** + +Run: `cd apps/web && npx tsc --noEmit` +Expected: no errors. + +- [ ] **Step 4: Commit** + +```bash +git add apps/web/app/page.tsx +git commit -m "feat(web): whiteboard photo upload entry on home page" +``` + +--- + +## Task 9: Web — `/confirm` editable confirm screen + +**Files:** +- Create: `apps/web/app/confirm/page.tsx` + +- [ ] **Step 1: Create the confirm page** + +Create `apps/web/app/confirm/page.tsx`: + +```tsx +"use client"; + +import { useRouter } from "next/navigation"; +import { useEffect, useState } from "react"; +import { startAnalysis } from "@/lib/api"; +import type { DraftGraph } from "@/lib/types"; + +export default function Confirm() { + const router = useRouter(); + const [draft, setDraft] = useState(null); + const [loading, setLoading] = useState(false); + const [error, setError] = useState(null); + + useEffect(() => { + const raw = sessionStorage.getItem("zeroforce.draft"); + if (raw) setDraft(JSON.parse(raw)); + }, []); + + if (!draft) { + return ( +
+

No detected graph found.

+ +
+ ); + } + + function renameNode(id: string, label: string) { + setDraft((d) => d && { ...d, nodes: d.nodes.map((n) => (n.id === id ? { ...n, label } : n)) }); + } + function deleteNode(id: string) { + setDraft((d) => + d && { + ...d, + nodes: d.nodes.filter((n) => n.id !== id), + edges: d.edges.filter((e) => e.source !== id && e.target !== id), + focal_node: d.focal_node === id ? "" : d.focal_node, + }, + ); + } + function addNode() { + setDraft((d) => { + if (!d) return d; + const id = `node_${d.nodes.length + 1}`; + return { ...d, nodes: [...d.nodes, { id, label: "New node" }] }; + }); + } + function deleteEdge(i: number) { + setDraft((d) => d && { ...d, edges: d.edges.filter((_, idx) => idx !== i) }); + } + function reverseEdge(i: number) { + setDraft( + (d) => + d && { + ...d, + edges: d.edges.map((e, idx) => (idx === i ? { source: e.target, target: e.source } : e)), + }, + ); + } + function addEdge() { + setDraft((d) => { + if (!d || d.nodes.length < 2) return d; + return { ...d, edges: [...d.edges, { source: d.nodes[0].id, target: d.nodes[1].id }] }; + }); + } + + async function analyze() { + if (!draft || !draft.focal_node) { + setError("Pick a focal node first."); + return; + } + setLoading(true); + setError(null); + try { + const focalLabel = draft.nodes.find((n) => n.id === draft.focal_node)?.label ?? "whiteboard graph"; + const { id } = await startAnalysis(focalLabel, "company", draft); + router.push(`/analyze/${id}`); + } catch (err) { + setError((err as Error).message); + setLoading(false); + } + } + + return ( +
+

Confirm the detected graph

+

+ Fix any misread labels or arrows, choose the focal node, then analyze. +

+ +
+
+

Nodes

+ +
+
+ {draft.nodes.map((n) => ( +
+ setDraft((d) => d && { ...d, focal_node: n.id })} + title="focal node" + /> + renameNode(n.id, e.target.value)} + className="flex-1 rounded border border-edge bg-panel px-2 py-1" + /> + +
+ ))} +
+
+ +
+
+

Edges (supplier → customer)

+ +
+
+ {draft.edges.map((e, i) => ( +
+ + + + + +
+ ))} +
+
+ + {error &&

{error}

} + +
+ + +
+
+ ); +} +``` + +- [ ] **Step 2: Type-check + build** + +Run: `cd apps/web && npx tsc --noEmit && npm run build` +Expected: no type errors; build succeeds (the `/confirm` route compiles). + +- [ ] **Step 3: Commit** + +```bash +git add apps/web/app/confirm/page.tsx +git commit -m "feat(web): editable confirm screen for detected whiteboard graph" +``` + +--- + +## Task 10: Docs + infra Gemini-only sweep + +**Files:** +- Modify: `README.md`, `.env.example`, `infra/docker-compose.dev.yml`, `infra/terraform/variables.tf` (and any other terraform file referencing `dev`/Cerebras/Tavily) + +- [ ] **Step 1: Find every stale reference** + +Run: `grep -rin "cerebras\|tavily\|ZEROFORCE_MODE=dev\|\bdev\b.*mode\|fake.*dev.*prod" README.md .env.example infra/` +Expected: a list of lines to fix. + +- [ ] **Step 2: Update the README modes section** + +In `README.md`, replace the three-mode table and surrounding prose (the "Modes (the dev/prod hinge)" section) with a Gemini-only description: one runtime (Gemini via Vertex / ADC, built-in Google Search grounding), and a note that `ZEROFORCE_MODE=fake` exists only for offline tests/CI. Remove the `dev` row, the Cerebras/Tavily columns, and `CEREBRAS_*`/`TAVILY_*` env references. Update the `make install` line to drop "Cerebras/Tavily". + +- [ ] **Step 3: Update `.env.example`** + +Remove `CEREBRAS_API_KEY`, `CEREBRAS_MODEL`, `CEREBRAS_TEXT_MODEL`, `TAVILY_API_KEY`. Set `ZEROFORCE_MODE=prod` as the documented default and keep the GCP/Gemini vars (`GCP_PROJECT`, `GCP_REGION`, `GEMINI_MODEL`). + +- [ ] **Step 4: Update infra** + +In `infra/docker-compose.dev.yml` and `infra/terraform/*`, remove `CEREBRAS_*` / `TAVILY_*` env entries and any `ZEROFORCE_MODE=dev` defaults; set `ZEROFORCE_MODE=prod`. Leave service topology otherwise unchanged. + +- [ ] **Step 5: Verify no stale references remain** + +Run: `grep -rin "cerebras\|tavily" README.md .env.example infra/ packages/ services/` +Expected: no matches (test files and code already cleaned in earlier tasks). + +- [ ] **Step 6: Commit** + +```bash +git add README.md .env.example infra/ +git commit -m "docs(infra): Gemini-only sweep — drop dev/Cerebras/Tavily" +``` + +--- + +## Task 11: Full verification + +- [ ] **Step 1: Backend + engine + providers test suites** + +Run: `cd packages/zeroforce && uv run pytest -q` +Expected: PASS (engine/oracle gates unchanged). + +Run: `cd packages/providers && uv run pytest -q` +Expected: PASS. + +Run: `cd services/backend && uv run pytest -q` +Expected: PASS — including `test_pipeline_integration.py` (verify it does not pin `mode="dev"`; if it does, update the fixture to `mode="fake"`). + +- [ ] **Step 2: Frontend** + +Run: `cd apps/web && npx tsc --noEmit && npm run build` +Expected: no type errors; build succeeds. + +- [ ] **Step 3: Manual Gemini verification (requires GCP ADC)** + +This is the only path not covered offline. With `ZEROFORCE_MODE=prod` and ADC configured: +1. `make dev`, open `http://localhost:3000`. +2. Click "📷 Upload a photo of a whiteboard"; pick a photo of a hand-drawn supplier graph. +3. Confirm the detected nodes/edges look right (correct any misreads); pick the focal node; click Analyze. +4. Verify it lands on `/analyze/[id]` with a researched, weighted graph whose topology matches the drawing. + +Record the outcome (pass/fail + any misreads) in `PROGRESS.md`. + +- [ ] **Step 4: Final commit (only if Step 1-2 required fixups)** + +```bash +git add -A +git commit -m "test: align suites with Gemini-only runtime" +``` + +--- + +## Self-review notes + +- **Spec coverage:** Part A (collapse) → Tasks 2 + 10; vision Stage 1 → Tasks 1, 3, 6; confirm screen → Tasks 8, 9; Stage 3 enrich → Tasks 4, 5; tests → every task + Task 11. All spec sections mapped. +- **Type consistency:** `DraftGraph`/`DraftNode`/`DraftEdge`, `extract_structure_from_image(image_bytes, mime_type)`, `seed`/`seed_graph`, and the `SEED_STRUCTURE_JSON:` marker are used identically across Python and TS tasks. +- **Known ordering note:** Task 2 Step 5 adds `seed=seed` to the extractor before `extraction_user` gains the parameter in Task 4 — execute Task 4 right after Task 2 (do not run the full backend suite between them). From a5662f153f0262c07c5e4f57b202d72e7188b0a4 Mon Sep 17 00:00:00 2001 From: justin06lee Date: Fri, 22 May 2026 15:59:26 -0700 Subject: [PATCH 03/18] feat(types): add DraftGraph for whiteboard vision structure --- packages/zeroforce/tests/test_draftgraph.py | 13 ++++++++++++ packages/zeroforce/zeroforce/types.py | 22 +++++++++++++++++++++ 2 files changed, 35 insertions(+) create mode 100644 packages/zeroforce/tests/test_draftgraph.py diff --git a/packages/zeroforce/tests/test_draftgraph.py b/packages/zeroforce/tests/test_draftgraph.py new file mode 100644 index 0000000..ffba451 --- /dev/null +++ b/packages/zeroforce/tests/test_draftgraph.py @@ -0,0 +1,13 @@ +from zeroforce.types import DraftEdge, DraftGraph, DraftNode + + +def test_draftgraph_roundtrips(): + dg = DraftGraph( + nodes=[DraftNode(id="apple", label="Apple"), DraftNode(id="tsmc", label="TSMC")], + edges=[DraftEdge(source="tsmc", target="apple")], + focal_node="apple", + ) + again = DraftGraph.model_validate_json(dg.model_dump_json()) + assert again.focal_node == "apple" + assert [n.id for n in again.nodes] == ["apple", "tsmc"] + assert again.edges[0].source == "tsmc" diff --git a/packages/zeroforce/zeroforce/types.py b/packages/zeroforce/zeroforce/types.py index 4eacf71..9696703 100644 --- a/packages/zeroforce/zeroforce/types.py +++ b/packages/zeroforce/zeroforce/types.py @@ -31,6 +31,28 @@ class Edge(BaseModel): asof: date | None = None +class DraftNode(BaseModel): + """A node detected from a hand-drawn whiteboard graph (structure only, pre-research).""" + + id: str + label: str + + +class DraftEdge(BaseModel): + """A directed arrow detected from a whiteboard graph; source/target reference DraftNode.id.""" + + source: str + target: str + + +class DraftGraph(BaseModel): + """Vision output: the boxes + arrows read off a whiteboard, before research enrichment.""" + + nodes: list[DraftNode] + edges: list[DraftEdge] + focal_node: str + + class Graph(BaseModel): nodes: list[Node] edges: list[Edge] From f425e8377c8e223e494e4790d5f5cb568a89920c Mon Sep 17 00:00:00 2001 From: justin06lee Date: Fri, 22 May 2026 16:02:22 -0700 Subject: [PATCH 04/18] feat(runtime): collapse to Gemini-only; keep fake as test double --- packages/providers/pyproject.toml | 3 +- packages/providers/tests/test_modes.py | 20 +++ .../providers/zeroforce_providers/cerebras.py | 137 ------------------ .../providers/zeroforce_providers/factory.py | 34 ++--- .../providers/zeroforce_providers/tavily.py | 36 ----- services/backend/zeroforce_backend/config.py | 4 +- .../zeroforce_backend/extractor/core.py | 36 ++--- 7 files changed, 46 insertions(+), 224 deletions(-) create mode 100644 packages/providers/tests/test_modes.py delete mode 100644 packages/providers/zeroforce_providers/cerebras.py delete mode 100644 packages/providers/zeroforce_providers/tavily.py diff --git a/packages/providers/pyproject.toml b/packages/providers/pyproject.toml index 068c295..a387af2 100644 --- a/packages/providers/pyproject.toml +++ b/packages/providers/pyproject.toml @@ -1,13 +1,12 @@ [project] name = "zeroforce-providers" version = "0.0.1" -description = "LLM + search provider abstraction for zeroforce (fake/dev/prod)." +description = "LLM + search provider abstraction for zeroforce (fake/prod)." requires-python = ">=3.12,<3.13" dependencies = ["pydantic>=2.7,<3.0", "zeroforce", "zeroforce-schemas"] [project.optional-dependencies] # Heavy vendor SDKs are optional; `fake` mode needs none of them. -dev = ["openai>=1.40", "tavily-python>=0.5"] prod = ["google-genai>=0.3"] [build-system] diff --git a/packages/providers/tests/test_modes.py b/packages/providers/tests/test_modes.py new file mode 100644 index 0000000..4150523 --- /dev/null +++ b/packages/providers/tests/test_modes.py @@ -0,0 +1,20 @@ +import pytest + +from zeroforce_providers.factory import get_mode, make_llm, make_search +from zeroforce_providers.fake import FakeLLMProvider, FakeSearchProvider + + +def test_dev_mode_rejected(monkeypatch): + monkeypatch.setenv("ZEROFORCE_MODE", "dev") + with pytest.raises(ValueError): + get_mode() + + +def test_default_mode_is_prod(monkeypatch): + monkeypatch.delenv("ZEROFORCE_MODE", raising=False) + assert get_mode() == "prod" + + +def test_fake_mode_returns_fakes(): + assert isinstance(make_llm("fake"), FakeLLMProvider) + assert isinstance(make_search("fake"), FakeSearchProvider) diff --git a/packages/providers/zeroforce_providers/cerebras.py b/packages/providers/zeroforce_providers/cerebras.py deleted file mode 100644 index d892317..0000000 --- a/packages/providers/zeroforce_providers/cerebras.py +++ /dev/null @@ -1,137 +0,0 @@ -"""Dev LLM provider: Cerebras inference via the OpenAI-compatible endpoint. - -Cerebras's `https://api.cerebras.ai/v1` is OpenAI-compatible, so it reuses the OpenAI client. -Structured extraction uses JSON-object mode plus an explicit JSON-Schema instruction (portable -across models), then validates against the Pydantic schema with a brace-extraction fallback. -""" - -from __future__ import annotations - -import json -import os -import re - -_BASE_URL = "https://api.cerebras.ai/v1" - -_JSON_ONLY_SYSTEM = ( - " You output ONLY a single raw JSON object. No prose, no explanation, no markdown code " - "fences, no tags. Your entire response must start with '{' and end with '}'." -) - - -class CerebrasProvider: - """Frugal per-purpose model selection (override via env): - - structured extraction (accuracy-critical) -> CEREBRAS_MODEL, default zai-glm-4.7 - - text report (low-stakes prose) -> CEREBRAS_TEXT_MODEL, default gpt-oss-120b - """ - - name = "cerebras" - - def __init__(self, model: str | None = None, text_model: str | None = None): - from openai import AsyncOpenAI # lazy: optional dep (OpenAI-compatible client) - - self.model = model or os.environ.get("CEREBRAS_MODEL", "zai-glm-4.7") - self.text_model = text_model or os.environ.get("CEREBRAS_TEXT_MODEL", "gpt-oss-120b") - self.name = f"cerebras/{self.model}+{self.text_model}" - self.client = AsyncOpenAI( - base_url=os.environ.get("CEREBRAS_BASE_URL", _BASE_URL), - api_key=os.environ["CEREBRAS_API_KEY"], - ) - - async def generate_structured(self, system, user, schema, temperature=0.2, max_output_tokens=16384): - schema_json = json.dumps(schema.model_json_schema()) - sys_msg = system + _JSON_ONLY_SYSTEM - user_msg = f"{user}\n\nReturn ONLY a JSON object matching this JSON Schema:\n{schema_json}" - - # Reasoning models (e.g. zai-glm-4.7) sometimes emit or prose and ignore - # json_object. Try progressively stricter, then fall back to the lighter text model. - attempts = [ - (self.model, temperature, True), - (self.model, 0.0, False), - (self.text_model, 0.0, True), - ] - last_content = "" - diagnostics: list[str] = [] - for model, temp, json_mode in attempts: - try: - content, meta = await self._chat(sys_msg, user_msg, model, temp, max_output_tokens, json_mode) - except Exception as e: # surface the real cause (rate limit, bad model id, etc.) - diagnostics.append(f"{model}: API error: {type(e).__name__}: {e}") - continue - last_content = content or last_content - cleaned = _clean_json_text(content) - if not cleaned: - diagnostics.append(f"{model}: empty content ({meta})") - continue - try: - return schema.model_validate_json(cleaned) - except Exception: - try: - return schema.model_validate_json(_extract_json(cleaned)) - except Exception as e: - diagnostics.append(f"{model}: unparseable ({meta}): {type(e).__name__}") - - raise ValueError( - "Cerebras returned no parseable JSON after retries + fallback. " - f"Last response (truncated): {last_content[:300]!r}. Attempts: " + " | ".join(diagnostics) - ) - - async def _chat(self, system, user, model, temperature, max_tokens, json_mode) -> tuple[str, str]: - kwargs = {} - if json_mode: - kwargs["response_format"] = {"type": "json_object"} - resp = await self.client.chat.completions.create( - model=model, - messages=[{"role": "system", "content": system}, {"role": "user", "content": user}], - temperature=temperature, - max_tokens=max_tokens, - **kwargs, - ) - choice = resp.choices[0] - msg = choice.message - content = msg.content or "" - # Reasoning models may stash the answer in a reasoning field and truncate before - # emitting content (finish_reason=length). Recover it and report finish_reason. - if not content: - content = getattr(msg, "reasoning_content", None) or getattr(msg, "reasoning", None) or "" - meta = f"finish={choice.finish_reason}" - return content, meta - - async def generate_text(self, system, user, temperature=0.4, max_output_tokens=1024) -> str: - resp = await self.client.chat.completions.create( - model=self.text_model, # cheaper model for low-stakes prose (frugal) - messages=[ - {"role": "system", "content": system}, - {"role": "user", "content": user}, - ], - temperature=temperature, - max_tokens=max_output_tokens, - ) - return resp.choices[0].message.content or "" - - async def health(self) -> bool: - try: - await self.client.models.list() - return True - except Exception: - return False - - -def _clean_json_text(text: str) -> str: - """Strip reasoning blocks and markdown code fences before parsing.""" - if not text: - return "" - text = re.sub(r".*?", "", text, flags=re.DOTALL | re.IGNORECASE) - text = re.sub(r"", "", text, flags=re.IGNORECASE) # unclosed think tag - text = re.sub(r"```(?:json)?\s*", "", text) - text = text.replace("```", "") - return text.strip() - - -def _extract_json(text: str) -> str: - """Best-effort: pull the outermost JSON object from a response with stray prose/fences.""" - start = text.find("{") - end = text.rfind("}") - if start == -1 or end == -1 or end < start: - raise ValueError("no JSON object found in model response") - return text[start : end + 1] diff --git a/packages/providers/zeroforce_providers/factory.py b/packages/providers/zeroforce_providers/factory.py index df25aaa..c7f0fcf 100644 --- a/packages/providers/zeroforce_providers/factory.py +++ b/packages/providers/zeroforce_providers/factory.py @@ -1,9 +1,8 @@ -"""Provider factory — the dev/prod/fake hinge (spec §3.2, §6.6). +"""Provider factory — Gemini-only runtime with a fake test double. Mode is a single env var `ZEROFORCE_MODE`: - fake (default) -> deterministic offline providers; no keys, no network. - dev -> Cerebras (CEREBRAS_API_KEY) + Tavily (TAVILY_API_KEY). - prod -> Gemini 3.5 Flash (Vertex, GCP ADC) + Vertex grounding. + prod (default) -> Gemini (Vertex, GCP ADC); web grounding is built into the provider. + fake -> deterministic offline providers; set only by tests/CI. No keys, no network. """ from __future__ import annotations @@ -13,13 +12,13 @@ from .base import LLMProvider, SearchProvider -Mode = Literal["fake", "dev", "prod"] +Mode = Literal["fake", "prod"] def get_mode() -> Mode: - mode = os.environ.get("ZEROFORCE_MODE", "fake").lower() - if mode not in ("fake", "dev", "prod"): - raise ValueError(f"ZEROFORCE_MODE must be fake|dev|prod, got {mode!r}") + mode = os.environ.get("ZEROFORCE_MODE", "prod").lower() + if mode not in ("fake", "prod"): + raise ValueError(f"ZEROFORCE_MODE must be fake|prod, got {mode!r}") return mode # type: ignore[return-value] @@ -29,25 +28,16 @@ def make_llm(mode: Mode | None = None) -> LLMProvider: from .fake import FakeLLMProvider return FakeLLMProvider() - if mode == "dev": - from .cerebras import CerebrasProvider - - return CerebrasProvider() from .gemini import GeminiProvider return GeminiProvider() def make_search(mode: Mode | None = None) -> SearchProvider: + """Only the fake double needs a standalone search provider; in prod, web grounding lives + inside the Gemini provider (built-in Google Search), so this returns a fake placeholder + that the prod extraction path never calls.""" mode = mode or get_mode() - if mode == "fake": - from .fake import FakeSearchProvider - - return FakeSearchProvider() - if mode == "dev": - from .tavily import TavilyProvider - - return TavilyProvider() - from .vertex import VertexGroundingProvider + from .fake import FakeSearchProvider - return VertexGroundingProvider() + return FakeSearchProvider() diff --git a/packages/providers/zeroforce_providers/tavily.py b/packages/providers/zeroforce_providers/tavily.py deleted file mode 100644 index de3f677..0000000 --- a/packages/providers/zeroforce_providers/tavily.py +++ /dev/null @@ -1,36 +0,0 @@ -"""Dev search provider: Tavily (spec §6.5).""" - -from __future__ import annotations - -import os - -from .base import SearchResult - - -class TavilyProvider: - name = "tavily" - - def __init__(self): - from tavily import AsyncTavilyClient # lazy: optional dep - - self.client = AsyncTavilyClient(api_key=os.environ["TAVILY_API_KEY"]) - - async def search(self, query, max_results=5, include_domains=None, recency_days=None): - resp = await self.client.search( - query=query, - search_depth="advanced", - max_results=max_results, - include_domains=include_domains or [], - days=recency_days, - include_answer=False, - include_raw_content=False, - ) - return [ - SearchResult(title=r["title"], url=r["url"], snippet=r["content"], score=r.get("score")) - for r in resp["results"] - ] - - async def fetch(self, url: str) -> str: - resp = await self.client.extract(urls=[url]) - results = resp.get("results", []) - return results[0]["raw_content"] if results else "" diff --git a/services/backend/zeroforce_backend/config.py b/services/backend/zeroforce_backend/config.py index 47c18e1..15ba6c2 100644 --- a/services/backend/zeroforce_backend/config.py +++ b/services/backend/zeroforce_backend/config.py @@ -24,7 +24,7 @@ def _load_dotenv_files() -> None: @dataclass(frozen=True) class Settings: - mode: str # fake | dev | prod + mode: str # prod (Gemini) | fake (tests only) db_path: str # SQLite file (dev) — Firestore in prod is selected separately default_max_nodes: int = 12 hard_max_nodes: int = 18 @@ -39,7 +39,7 @@ class Settings: def load_settings() -> Settings: _load_dotenv_files() return Settings( - mode=os.environ.get("ZEROFORCE_MODE", "fake").lower(), + mode=os.environ.get("ZEROFORCE_MODE", "prod").lower(), db_path=os.environ.get("ZEROFORCE_DB", "./data/zeroforce.db"), api_token=os.environ.get("ZEROFORCE_API_TOKEN") or None, ) diff --git a/services/backend/zeroforce_backend/extractor/core.py b/services/backend/zeroforce_backend/extractor/core.py index 9eaf14f..d14a188 100644 --- a/services/backend/zeroforce_backend/extractor/core.py +++ b/services/backend/zeroforce_backend/extractor/core.py @@ -1,15 +1,16 @@ -"""Extractor service core — Prompt 1 graph extraction with grounding (spec §3.3 step 4, §8.1). +"""Extractor service core — graph extraction with grounding (spec §3.3 step 4, §8.1). -- fake: one structured call (the fake provider builds the graph; search ignored). -- dev: one Tavily search, snippets concatenated into the prompt, then one Cerebras structured call. -- prod: one Gemini call whose two-pass grounding happens inside the provider. +- prod: one Gemini call whose two-pass grounding (Google Search) happens inside the provider. +- fake: one structured call (the fake provider builds the graph deterministically). -Shape is identical across modes (ground -> structure); only the mechanism differs. +When `seed` is provided (a whiteboard DraftGraph), it is embedded in the prompt as the +authoritative skeleton and the model researches the named entities to fill weights, tiers, +sectors, and citations rather than inventing a fresh topology. """ from __future__ import annotations -from zeroforce.types import Graph +from zeroforce.types import DraftGraph, Graph from zeroforce_providers import LLMProvider, SearchProvider from ..prompts import EXTRACTION_SYSTEM, extraction_user @@ -23,25 +24,10 @@ async def extract_graph( llm: LLMProvider, search: SearchProvider, correction: str = "", + seed: DraftGraph | None = None, ) -> Graph: - grounding = "" - if mode == "dev": - # Research several tiers so the model can build depth + redundancy, not a flat star. - queries = [ - f"{user_input} tier 1 suppliers list 10-K", - f"{user_input} tier 2 component suppliers and sub-suppliers", - f"{user_input} raw materials and single-source supplier risk", - ] - seen: set[str] = set() - lines: list[str] = [] - for q in queries: - for r in await search.search(q, max_results=4): - if r.url in seen: - continue - seen.add(r.url) - lines.append(f"- {r.title}: {r.snippet} ({r.url})") - grounding = "\n".join(lines) - - user = extraction_user(user_input, max_nodes=max_nodes, grounding=grounding, correction=correction) + user = extraction_user( + user_input, max_nodes=max_nodes, grounding="", correction=correction, seed=seed + ) graph = await llm.generate_structured(EXTRACTION_SYSTEM, user, Graph, temperature=0.2) return graph From 72984c66529e2113c655aab5d63afd10a3c12aff Mon Sep 17 00:00:00 2001 From: justin06lee Date: Fri, 22 May 2026 16:07:26 -0700 Subject: [PATCH 05/18] feat(extractor): seed-aware extraction from whiteboard DraftGraph Co-Authored-By: Claude Opus 4.7 (1M context) --- .../providers/zeroforce_providers/fake.py | 60 ++++++++++++++++++- .../backend/tests/test_seed_extraction.py | 35 +++++++++++ services/backend/zeroforce_backend/prompts.py | 21 ++++++- 3 files changed, 113 insertions(+), 3 deletions(-) create mode 100644 services/backend/tests/test_seed_extraction.py diff --git a/packages/providers/zeroforce_providers/fake.py b/packages/providers/zeroforce_providers/fake.py index c3a4ce6..836412d 100644 --- a/packages/providers/zeroforce_providers/fake.py +++ b/packages/providers/zeroforce_providers/fake.py @@ -15,7 +15,7 @@ from pydantic import BaseModel -from zeroforce.types import Edge, Graph, Node +from zeroforce.types import DraftGraph, Edge, Graph, Node from .base import SearchResult @@ -128,6 +128,60 @@ def link(src: str, dst: str, conf: str, tier_label: str): ) +def build_fake_graph_from_seed(seed: DraftGraph) -> Graph: + """Turn a whiteboard DraftGraph into a full Graph deterministically, preserving the drawn + topology. Tiers are BFS distance (in reversed edges) from the focal sink; incoming weights + per target are normalized to sum to 1.0 (first edge absorbs the remainder).""" + ids = [n.id for n in seed.nodes] + label = {n.id: n.label for n in seed.nodes} + + # tier = shortest distance from focal walking edges backwards (supplier side deeper). + incoming: dict[str, list[str]] = {i: [] for i in ids} + for e in seed.edges: + if e.target in incoming and e.source in incoming: + incoming[e.target].append(e.source) + tier: dict[str, int] = {seed.focal_node: 0} + frontier = [seed.focal_node] + while frontier: + nxt: list[str] = [] + for node in frontier: + for supplier in incoming.get(node, []): + if supplier not in tier: + tier[supplier] = tier[node] + 1 + nxt.append(supplier) + frontier = nxt + for i in ids: + tier.setdefault(i, 1) + + sector_for = {0: "focal", 1: "assembly", 2: "components", 3: "materials"} + nodes = [ + Node(id=i, label=label[i], sector=sector_for.get(min(tier[i], 3), "materials"), tier=tier[i]) + for i in ids + ] + + by_target: dict[str, list[str]] = {} + for e in seed.edges: + if e.target in incoming and e.source in incoming: + by_target.setdefault(e.target, []).append(e.source) + + edges: list[Edge] = [] + for dst, sources in by_target.items(): + n = len(sources) + base = 1.0 / n + for idx, src in enumerate(sources): + w = (1.0 - base * (n - 1)) if idx == 0 else base + edges.append(Edge( + source=src, target=dst, weight=w, confidence="medium", + citation="whiteboard (fake seed)", + reasoning=f"{label.get(src, src)} supplies {label.get(dst, dst)} (drawn).", + )) + + return Graph( + nodes=nodes, edges=edges, focal_node=seed.focal_node, + extraction_model="fake/seed-v1", extraction_timestamp=_FIXED_TS, + ) + + class FakeLLMProvider: name = "fake/deterministic-v1" @@ -136,6 +190,10 @@ def __init__(self, max_nodes: int = 12): async def generate_structured(self, system, user, schema, temperature=0.2, **_): if schema is Graph: + marker = "SEED_STRUCTURE_JSON:" + if marker in user: + blob = user.split(marker, 1)[1].strip().splitlines()[0] + return build_fake_graph_from_seed(DraftGraph.model_validate_json(blob)) return build_fake_graph(user, max_nodes=self._max_nodes) # Generic fallback: instantiate the schema from a deterministic empty-ish payload. return _instantiate_minimal(schema) diff --git a/services/backend/tests/test_seed_extraction.py b/services/backend/tests/test_seed_extraction.py new file mode 100644 index 0000000..fd61db6 --- /dev/null +++ b/services/backend/tests/test_seed_extraction.py @@ -0,0 +1,35 @@ +import asyncio + +from zeroforce.types import DraftEdge, DraftGraph, DraftNode +from zeroforce_backend.prompts import extraction_user +from zeroforce_providers.fake import FakeLLMProvider +from zeroforce.types import Graph + + +def _seed(): + return DraftGraph( + nodes=[DraftNode(id="acme", label="Acme"), DraftNode(id="widgetco", label="WidgetCo")], + edges=[DraftEdge(source="widgetco", target="acme")], + focal_node="acme", + ) + + +def test_prompt_embeds_seed_marker(): + prompt = extraction_user("Acme", max_nodes=12, seed=_seed()) + assert "SEED_STRUCTURE_JSON:" in prompt + assert "widgetco" in prompt + + +def test_fake_preserves_seed_topology(): + prompt = extraction_user("Acme", max_nodes=12, seed=_seed()) + graph = asyncio.run(FakeLLMProvider().generate_structured("sys", prompt, Graph)) + assert graph.focal_node == "acme" + assert {n.id for n in graph.nodes} == {"acme", "widgetco"} + assert any(e.source == "widgetco" and e.target == "acme" for e in graph.edges) + + +def test_fake_without_seed_unchanged(): + prompt = extraction_user("Apple semiconductor", max_nodes=12) + graph = asyncio.run(FakeLLMProvider().generate_structured("sys", prompt, Graph)) + # No seed marker -> the normal deterministic multi-tier fake graph (tier-0 focal present). + assert any(n.tier == 0 for n in graph.nodes) diff --git a/services/backend/zeroforce_backend/prompts.py b/services/backend/zeroforce_backend/prompts.py index f93f3ff..1a04b26 100644 --- a/services/backend/zeroforce_backend/prompts.py +++ b/services/backend/zeroforce_backend/prompts.py @@ -2,6 +2,8 @@ from __future__ import annotations +from zeroforce.types import DraftGraph + EXTRACTION_SYSTEM = """\ You are a supply chain analyst with expertise in corporate procurement and SEC financial disclosures. Given a natural language description of a company, product, or supply chain, @@ -11,11 +13,26 @@ general knowledge. You explicitly mark estimated weights as such. You never invent citations.""" -def extraction_user(user_input: str, max_nodes: int, grounding: str = "", correction: str = "") -> str: +def extraction_user( + user_input: str, + max_nodes: int, + grounding: str = "", + correction: str = "", + seed: "DraftGraph | None" = None, +) -> str: grounding_block = f"\n\nResearched findings to ground the graph in real, named companies:\n{grounding}\n" if grounding else "" correction_block = f"\n\nCORRECTION — your previous attempt was rejected:\n{correction}\n" if correction else "" + seed_block = "" + if seed is not None: + seed_block = ( + "\n\nThe user drew this supply-chain structure on a whiteboard. Treat it as the " + "AUTHORITATIVE SKELETON: keep these exact nodes and the drawn arrow directions, " + "research each named entity to fill weights/confidence/tier/sector/citations, and " + "add only edges that are obviously implied. Do NOT invent a different topology.\n" + f"SEED_STRUCTURE_JSON:\n{seed.model_dump_json()}\n" + ) return f"""\ -Build a MULTI-TIER supply chain dependency graph for: {user_input}{grounding_block}{correction_block} +Build a MULTI-TIER supply chain dependency graph for: {user_input}{grounding_block}{correction_block}{seed_block} This must be a realistic, multi-level network — NOT a flat star where every supplier points straight at the focal firm. Trace the chain UPSTREAM through several tiers. From 987f3ffd56404a0e294fabe3d1db22f11c8d43dd Mon Sep 17 00:00:00 2001 From: justin06lee Date: Fri, 22 May 2026 16:11:18 -0700 Subject: [PATCH 06/18] fix(fake): fall back to generic build on seed-marker parse failure Co-Authored-By: Claude Opus 4.7 (1M context) --- packages/providers/zeroforce_providers/fake.py | 6 +++++- services/backend/tests/test_seed_extraction.py | 9 +++++++++ 2 files changed, 14 insertions(+), 1 deletion(-) diff --git a/packages/providers/zeroforce_providers/fake.py b/packages/providers/zeroforce_providers/fake.py index 836412d..b66ed7a 100644 --- a/packages/providers/zeroforce_providers/fake.py +++ b/packages/providers/zeroforce_providers/fake.py @@ -192,8 +192,12 @@ async def generate_structured(self, system, user, schema, temperature=0.2, **_): if schema is Graph: marker = "SEED_STRUCTURE_JSON:" if marker in user: + # model_dump_json() is single-line, so the seed payload is the line after the marker. blob = user.split(marker, 1)[1].strip().splitlines()[0] - return build_fake_graph_from_seed(DraftGraph.model_validate_json(blob)) + try: + return build_fake_graph_from_seed(DraftGraph.model_validate_json(blob)) + except Exception: # noqa: BLE001 - not a real seed payload; fall back to generic build + pass return build_fake_graph(user, max_nodes=self._max_nodes) # Generic fallback: instantiate the schema from a deterministic empty-ish payload. return _instantiate_minimal(schema) diff --git a/services/backend/tests/test_seed_extraction.py b/services/backend/tests/test_seed_extraction.py index fd61db6..43d1ce1 100644 --- a/services/backend/tests/test_seed_extraction.py +++ b/services/backend/tests/test_seed_extraction.py @@ -33,3 +33,12 @@ def test_fake_without_seed_unchanged(): graph = asyncio.run(FakeLLMProvider().generate_structured("sys", prompt, Graph)) # No seed marker -> the normal deterministic multi-tier fake graph (tier-0 focal present). assert any(n.tier == 0 for n in graph.nodes) + + +def test_fake_marker_collision_falls_back(): + # Prompt text contains the marker but no valid seed JSON -> graceful fallback, no crash. + from zeroforce.types import Graph + prompt = "Build a graph for: SEED_STRUCTURE_JSON: not-actually-json" + graph = asyncio.run(FakeLLMProvider().generate_structured("sys", prompt, Graph)) + assert isinstance(graph, Graph) + assert len(graph.nodes) >= 2 From 8c3b709977ffe836c083616501fe05c51f5c7582 Mon Sep 17 00:00:00 2001 From: justin06lee Date: Fri, 22 May 2026 16:13:10 -0700 Subject: [PATCH 07/18] feat(providers): add extract_structure_from_image vision method Co-Authored-By: Claude Opus 4.7 (1M context) --- packages/providers/tests/test_modes.py | 21 ++++++++++++ .../providers/zeroforce_providers/base.py | 8 +++++ .../providers/zeroforce_providers/fake.py | 20 +++++++++++- .../providers/zeroforce_providers/gemini.py | 32 +++++++++++++++++++ .../providers/zeroforce_providers/lazy.py | 3 ++ 5 files changed, 83 insertions(+), 1 deletion(-) diff --git a/packages/providers/tests/test_modes.py b/packages/providers/tests/test_modes.py index 4150523..799abe5 100644 --- a/packages/providers/tests/test_modes.py +++ b/packages/providers/tests/test_modes.py @@ -1,7 +1,11 @@ +import asyncio + import pytest +from zeroforce.types import DraftGraph from zeroforce_providers.factory import get_mode, make_llm, make_search from zeroforce_providers.fake import FakeLLMProvider, FakeSearchProvider +from zeroforce_providers.lazy import LazyLLM def test_dev_mode_rejected(monkeypatch): @@ -18,3 +22,20 @@ def test_default_mode_is_prod(monkeypatch): def test_fake_mode_returns_fakes(): assert isinstance(make_llm("fake"), FakeLLMProvider) assert isinstance(make_search("fake"), FakeSearchProvider) + + +def test_fake_vision_is_deterministic(): + llm = make_llm("fake") + png = b"\x89PNG\r\n\x1a\n" + b"fake-image-bytes" + a = asyncio.run(llm.extract_structure_from_image(png, "image/png")) + b = asyncio.run(llm.extract_structure_from_image(png, "image/png")) + assert isinstance(a, DraftGraph) + assert a.model_dump() == b.model_dump() + assert a.focal_node in {n.id for n in a.nodes} + assert len(a.nodes) >= 2 + + +def test_lazy_forwards_vision(): + llm = LazyLLM(lambda: make_llm("fake"), label="fake") + dg = asyncio.run(llm.extract_structure_from_image(b"x", "image/png")) + assert isinstance(dg, DraftGraph) diff --git a/packages/providers/zeroforce_providers/base.py b/packages/providers/zeroforce_providers/base.py index 8cb19c7..1c307ad 100644 --- a/packages/providers/zeroforce_providers/base.py +++ b/packages/providers/zeroforce_providers/base.py @@ -7,6 +7,8 @@ from pydantic import BaseModel +from zeroforce.types import DraftGraph + T = TypeVar("T", bound=BaseModel) @@ -39,6 +41,12 @@ async def generate_text( max_output_tokens: int = 1024, ) -> str: ... + async def extract_structure_from_image( + self, + image_bytes: bytes, + mime_type: str, + ) -> DraftGraph: ... + async def health(self) -> bool: ... diff --git a/packages/providers/zeroforce_providers/fake.py b/packages/providers/zeroforce_providers/fake.py index b66ed7a..4cc77ae 100644 --- a/packages/providers/zeroforce_providers/fake.py +++ b/packages/providers/zeroforce_providers/fake.py @@ -15,7 +15,7 @@ from pydantic import BaseModel -from zeroforce.types import DraftGraph, Edge, Graph, Node +from zeroforce.types import DraftEdge, DraftGraph, DraftNode, Edge, Graph, Node from .base import SearchResult @@ -182,6 +182,21 @@ def build_fake_graph_from_seed(seed: DraftGraph) -> Graph: ) +def build_fake_draft(image_bytes: bytes) -> DraftGraph: + """Deterministic whiteboard-vision stub: a tiny fixed 3-node draft, independent of the + bytes' content, so vision tests are byte-stable offline.""" + nodes = [ + DraftNode(id="focal", label="Focal Co"), + DraftNode(id="supplier_a", label="Supplier A"), + DraftNode(id="material_x", label="Material X"), + ] + edges = [ + DraftEdge(source="supplier_a", target="focal"), + DraftEdge(source="material_x", target="supplier_a"), + ] + return DraftGraph(nodes=nodes, edges=edges, focal_node="focal") + + class FakeLLMProvider: name = "fake/deterministic-v1" @@ -216,6 +231,9 @@ async def generate_text(self, system, user, temperature=0.4, **_) -> str: f"continuously. (This report was produced by the deterministic fake provider.)" ) + async def extract_structure_from_image(self, image_bytes, mime_type) -> DraftGraph: + return build_fake_draft(image_bytes) + async def health(self) -> bool: return True diff --git a/packages/providers/zeroforce_providers/gemini.py b/packages/providers/zeroforce_providers/gemini.py index fa8ba46..4fa64c2 100644 --- a/packages/providers/zeroforce_providers/gemini.py +++ b/packages/providers/zeroforce_providers/gemini.py @@ -17,6 +17,8 @@ import json import os +from zeroforce.types import DraftGraph + _STRUCTURE_RETRIES = 3 @@ -107,6 +109,36 @@ async def generate_text(self, system, user, temperature=0.4, max_output_tokens=1 ) return resp.text or "" + async def extract_structure_from_image(self, image_bytes, mime_type) -> DraftGraph: + """Read a hand-drawn supply-chain graph off a photo into a DraftGraph (structure only). + + Single multimodal call: image + instruction, response constrained to DraftGraph. No web + research here — that happens later in the seed-aware extraction pass. + """ + from google.genai.types import Part + + system = ( + "You read hand-drawn supply-chain diagrams from photos. Each box/label is a node; " + "each arrow is a directed edge pointing from supplier to customer. Transcribe the " + "drawing literally into the DraftGraph schema. Use lowercase slug ids derived from " + "labels (e.g. 'TSMC' -> 'tsmc'). Pick the most-downstream box (the one most arrows " + "point toward, or labelled as the focal firm/product) as focal_node. Do not invent " + "nodes or edges that are not drawn." + ) + image_part = Part.from_bytes(data=image_bytes, mime_type=mime_type) + cfg = self._cfg( + system_instruction=system, + temperature=0.0, + response_mime_type="application/json", + response_schema=DraftGraph, + ) + resp = await self.client.aio.models.generate_content( + model=self.model, + contents=[image_part, "Transcribe this whiteboard supply-chain drawing."], + config=cfg, + ) + return DraftGraph.model_validate_json(resp.text) + async def health(self) -> bool: try: await self.client.aio.models.generate_content(model=self.model, contents="ping") diff --git a/packages/providers/zeroforce_providers/lazy.py b/packages/providers/zeroforce_providers/lazy.py index 82ed16c..877cc89 100644 --- a/packages/providers/zeroforce_providers/lazy.py +++ b/packages/providers/zeroforce_providers/lazy.py @@ -33,6 +33,9 @@ async def generate_structured(self, *a, **k): async def generate_text(self, *a, **k): return await self._get().generate_text(*a, **k) + async def extract_structure_from_image(self, *a, **k): + return await self._get().extract_structure_from_image(*a, **k) + async def health(self) -> bool: try: return await self._get().health() From 35f7af4669c315a1883fa550a2920ce7c2c82a05 Mon Sep 17 00:00:00 2001 From: justin06lee Date: Fri, 22 May 2026 16:16:32 -0700 Subject: [PATCH 08/18] fix(gemini): guard empty vision response with a clear error --- packages/providers/zeroforce_providers/gemini.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/packages/providers/zeroforce_providers/gemini.py b/packages/providers/zeroforce_providers/gemini.py index 4fa64c2..552b917 100644 --- a/packages/providers/zeroforce_providers/gemini.py +++ b/packages/providers/zeroforce_providers/gemini.py @@ -137,6 +137,8 @@ async def extract_structure_from_image(self, image_bytes, mime_type) -> DraftGra contents=[image_part, "Transcribe this whiteboard supply-chain drawing."], config=cfg, ) + if not resp.text: + raise RuntimeError("Gemini returned no content for the whiteboard image (blocked or empty response).") return DraftGraph.model_validate_json(resp.text) async def health(self) -> bool: From d5156967509813eb906f97d23c09f72703327354 Mon Sep 17 00:00:00 2001 From: justin06lee Date: Fri, 22 May 2026 16:18:26 -0700 Subject: [PATCH 09/18] feat(pipeline): thread whiteboard seed_graph through analyze Co-Authored-By: Claude Opus 4.7 (1M context) --- packages/schemas/zeroforce_schemas/models.py | 3 ++- services/backend/tests/test_pipeline_retry.py | 2 +- .../backend/tests/test_seed_extraction.py | 20 +++++++++++++++++++ .../zeroforce_backend/orchestrator/core.py | 3 ++- .../zeroforce_backend/service_client.py | 13 ++++++------ .../backend/zeroforce_backend/worker_apps.py | 6 ++++-- 6 files changed, 36 insertions(+), 11 deletions(-) diff --git a/packages/schemas/zeroforce_schemas/models.py b/packages/schemas/zeroforce_schemas/models.py index 3885586..a6128ba 100644 --- a/packages/schemas/zeroforce_schemas/models.py +++ b/packages/schemas/zeroforce_schemas/models.py @@ -8,7 +8,7 @@ from pydantic import BaseModel, Field -from zeroforce.types import Graph, NodeAnalysis +from zeroforce.types import DraftGraph, Graph, NodeAnalysis AnalyzeMode = Literal["company", "sector", "product"] ConfidenceLevel = Literal["high", "medium", "low"] @@ -31,6 +31,7 @@ class AnalyzeRequest(BaseModel): enable_validation_pass: bool = True use_cached: bool = True wait: bool = False # sync 200 only honored when wait AND max_nodes <= 10 + seed_graph: DraftGraph | None = None # whiteboard-drawn skeleton; drives extraction when set class AnalysisResult(BaseModel): diff --git a/services/backend/tests/test_pipeline_retry.py b/services/backend/tests/test_pipeline_retry.py index dd05e25..ddf21af 100644 --- a/services/backend/tests/test_pipeline_retry.py +++ b/services/backend/tests/test_pipeline_retry.py @@ -56,7 +56,7 @@ def __init__(self, bad_attempts: int): self.bad_attempts = bad_attempts self.extract_calls: list[str] = [] - async def extract(self, user_input, max_nodes, correction=""): + async def extract(self, user_input, max_nodes, correction="", seed=None): self.extract_calls.append(correction) return _BAD if len(self.extract_calls) <= self.bad_attempts else _GOOD diff --git a/services/backend/tests/test_seed_extraction.py b/services/backend/tests/test_seed_extraction.py index 43d1ce1..5647000 100644 --- a/services/backend/tests/test_seed_extraction.py +++ b/services/backend/tests/test_seed_extraction.py @@ -42,3 +42,23 @@ def test_fake_marker_collision_falls_back(): graph = asyncio.run(FakeLLMProvider().generate_structured("sys", prompt, Graph)) assert isinstance(graph, Graph) assert len(graph.nodes) >= 2 + + +def test_analyze_request_accepts_seed(): + from zeroforce_schemas import AnalyzeRequest + + req = AnalyzeRequest(input="Acme", seed_graph=_seed()) + assert req.seed_graph is not None + assert req.seed_graph.focal_node == "acme" + + req2 = AnalyzeRequest(input="Acme") + assert req2.seed_graph is None + + +def test_inprocess_client_passes_seed(): + from zeroforce_backend.service_client import InProcessServiceClient + from zeroforce_providers.fake import FakeLLMProvider, FakeSearchProvider + + client = InProcessServiceClient("fake", FakeLLMProvider(), FakeSearchProvider()) + graph = asyncio.run(client.extract("Acme", 12, "", _seed())) + assert {n.id for n in graph.nodes} == {"acme", "widgetco"} diff --git a/services/backend/zeroforce_backend/orchestrator/core.py b/services/backend/zeroforce_backend/orchestrator/core.py index 99ba16e..a01d41f 100644 --- a/services/backend/zeroforce_backend/orchestrator/core.py +++ b/services/backend/zeroforce_backend/orchestrator/core.py @@ -61,6 +61,7 @@ def _job_id(self, req: AnalyzeRequest) -> str: "confidence_threshold": req.confidence_threshold, "enable_validation_pass": req.enable_validation_pass, "engine": self.settings.mode, + "seed": req.seed_graph.model_dump(mode="json") if req.seed_graph else None, } ) @@ -135,7 +136,7 @@ async def _run_pipeline(self, job_id: str, req: AnalyzeRequest) -> JobRecord: last_norm_err: NormalizationError | None = None for attempt in range(self.settings.max_extract_retries + 1): await self._set_stage(job_id, "extracting" if attempt == 0 else "re-extracting") - graph = await self.client.extract(req.input, req.max_nodes, correction) + graph = await self.client.extract(req.input, req.max_nodes, correction, req.seed_graph) await self._set_stage(job_id, "validating") try: diff --git a/services/backend/zeroforce_backend/service_client.py b/services/backend/zeroforce_backend/service_client.py index 33fb573..8104ffa 100644 --- a/services/backend/zeroforce_backend/service_client.py +++ b/services/backend/zeroforce_backend/service_client.py @@ -11,7 +11,7 @@ from typing import Protocol -from zeroforce.types import Graph, NodeAnalysis +from zeroforce.types import DraftGraph, Graph, NodeAnalysis from zeroforce_providers import LLMProvider, SearchProvider from .engine import analyze_graph @@ -21,7 +21,7 @@ class ServiceClient(Protocol): - async def extract(self, user_input: str, max_nodes: int, correction: str = "") -> Graph: ... + async def extract(self, user_input: str, max_nodes: int, correction: str = "", seed: DraftGraph | None = None) -> Graph: ... async def validate(self, graph: Graph, confidence_threshold: str) -> tuple[Graph, list[str]]: ... async def analyze(self, graph: Graph) -> tuple[list[NodeAnalysis], list[str], list[str]]: ... async def report(self, graph: Graph, analyses: list[NodeAnalysis]) -> str: ... @@ -33,10 +33,10 @@ def __init__(self, mode: str, llm: LLMProvider, search: SearchProvider): self.llm = llm self.search = search - async def extract(self, user_input: str, max_nodes: int, correction: str = "") -> Graph: + async def extract(self, user_input: str, max_nodes: int, correction: str = "", seed: DraftGraph | None = None) -> Graph: return await extract_graph( user_input, mode=self.mode, max_nodes=max_nodes, llm=self.llm, search=self.search, - correction=correction, + correction=correction, seed=seed, ) async def validate(self, graph, confidence_threshold): @@ -67,10 +67,11 @@ async def _post(self, service: str, path: str, payload: dict) -> dict: resp.raise_for_status() return resp.json() - async def extract(self, user_input: str, max_nodes: int, correction: str = "") -> Graph: + async def extract(self, user_input: str, max_nodes: int, correction: str = "", seed: DraftGraph | None = None) -> Graph: data = await self._post( "extractor", "/extract", - {"input": user_input, "max_nodes": max_nodes, "correction": correction}, + {"input": user_input, "max_nodes": max_nodes, "correction": correction, + "seed": seed.model_dump(mode="json") if seed is not None else None}, ) return Graph.model_validate(data) diff --git a/services/backend/zeroforce_backend/worker_apps.py b/services/backend/zeroforce_backend/worker_apps.py index 9543248..3331e1c 100644 --- a/services/backend/zeroforce_backend/worker_apps.py +++ b/services/backend/zeroforce_backend/worker_apps.py @@ -9,7 +9,7 @@ from fastapi import FastAPI -from zeroforce.types import Graph, NodeAnalysis +from zeroforce.types import DraftGraph, Graph, NodeAnalysis from zeroforce_providers import make_llm, make_search from .config import load_settings @@ -31,9 +31,11 @@ async def health(): @app.post("/extract") async def extract(payload: dict): + seed_raw = payload.get("seed") + seed = DraftGraph.model_validate(seed_raw) if seed_raw else None graph = await extract_graph( payload["input"], mode=settings.mode, max_nodes=payload.get("max_nodes", 12), - llm=llm, search=search, correction=payload.get("correction", ""), + llm=llm, search=search, correction=payload.get("correction", ""), seed=seed, ) return graph.model_dump(mode="json") From dcd03ecf779b80417b89092ada9bf2cf915af1a5 Mon Sep 17 00:00:00 2001 From: justin06lee Date: Fri, 22 May 2026 16:21:57 -0700 Subject: [PATCH 10/18] =?UTF-8?q?feat(api):=20POST=20/api/v1/vision=20?= =?UTF-8?q?=E2=80=94=20photo=20to=20DraftGraph?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.7 (1M context) --- services/backend/tests/test_vision.py | 34 +++++++++++++++++++ .../backend/zeroforce_backend/gateway/app.py | 17 ++++++++++ 2 files changed, 51 insertions(+) create mode 100644 services/backend/tests/test_vision.py diff --git a/services/backend/tests/test_vision.py b/services/backend/tests/test_vision.py new file mode 100644 index 0000000..183951c --- /dev/null +++ b/services/backend/tests/test_vision.py @@ -0,0 +1,34 @@ +import io + +from fastapi.testclient import TestClient + +from zeroforce_backend.config import Settings +from zeroforce_backend.gateway.app import build_context, create_app + + +def _client(): + ctx = build_context(Settings(mode="fake", db_path=":memory:")) + return TestClient(create_app(ctx)) + + +def test_vision_returns_draftgraph(): + client = _client() + png = b"\x89PNG\r\n\x1a\n" + b"drawn-graph" + resp = client.post( + "/api/v1/vision", + files={"file": ("board.png", io.BytesIO(png), "image/png")}, + ) + assert resp.status_code == 200 + body = resp.json() + assert "nodes" in body and "edges" in body and "focal_node" in body + assert body["focal_node"] in {n["id"] for n in body["nodes"]} + + +def test_vision_rejects_non_image(): + client = _client() + resp = client.post( + "/api/v1/vision", + files={"file": ("notes.txt", io.BytesIO(b"hello"), "text/plain")}, + ) + assert resp.status_code == 400 + assert resp.json()["detail"]["code"] == "invalid_input" diff --git a/services/backend/zeroforce_backend/gateway/app.py b/services/backend/zeroforce_backend/gateway/app.py index 2c0f649..4e6c391 100644 --- a/services/backend/zeroforce_backend/gateway/app.py +++ b/services/backend/zeroforce_backend/gateway/app.py @@ -38,6 +38,7 @@ VERSION = "0.1.0" _PII = re.compile(r"\b(\d{3}-\d{2}-\d{4}|\d{4}[ -]?\d{4}[ -]?\d{4}[ -]?\d{4})\b") +_IMAGE_MIMES = {"image/png", "image/jpeg", "image/webp", "image/heic", "image/heif"} @dataclass @@ -194,6 +195,22 @@ async def upload(file: UploadFile, _: None = Depends(require_auth)): redacted, n = _PII.subn("[REDACTED]", text) return {"filename": file.filename, "chars": len(redacted), "pii_redacted": n, "text": redacted[:20000]} + @app.post("/api/v1/vision") + async def vision(file: UploadFile, _: None = Depends(require_auth)): + mime = (file.content_type or "").lower() + if mime not in _IMAGE_MIMES: + raise HTTPException(status_code=400, detail={"code": "invalid_input", + "message": f"unsupported image type {mime!r}"}) + raw = await file.read() + if len(raw) > 20 * 1024 * 1024: + raise HTTPException(status_code=400, detail={"code": "invalid_input", "message": "file > 20MB"}) + try: + draft = await ctx.llm.extract_structure_from_image(raw, mime) + except Exception as e: # noqa: BLE001 - provider/parse failure -> clean 502 + raise HTTPException(status_code=502, detail={"code": "llm_provider_error", + "message": f"{type(e).__name__}: {e}"}) + return json.loads(draft.model_dump_json()) + return app From 6c44a3737449e92ec1733941278d445e8b56f0ce Mon Sep 17 00:00:00 2001 From: justin06lee Date: Fri, 22 May 2026 16:23:07 -0700 Subject: [PATCH 11/18] chore: sync uv.lock after dropping Cerebras/Tavily deps Co-Authored-By: Claude Opus 4.7 (1M context) --- uv.lock | 122 +------------------------------------------------------- 1 file changed, 1 insertion(+), 121 deletions(-) diff --git a/uv.lock b/uv.lock index 7b1a2e0..86e5d56 100644 --- a/uv.lock +++ b/uv.lock @@ -308,32 +308,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/cb/b1/3846dd7f199d53cb17f49cba7e651e9ce294d8497c8c150530ed11865bb8/iniconfig-2.3.0-py3-none-any.whl", hash = "sha256:f631c04d2c48c52b84d0d0549c99ff3859c98df65b3101406327ecc7d53fbf12", size = 7484, upload-time = "2025-10-18T21:55:41.639Z" }, ] -[[package]] -name = "jiter" -version = "0.15.0" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/66/b5/55f06bb281d92fb3cc86d14e1def2bd908bb77693183e7cb1f5a3c388b0c/jiter-0.15.0.tar.gz", hash = "sha256:4251acc80e2b7c9b7b8823456ea0fceeb0734dac2df7636d3c711b38476b5a76", size = 166640, upload-time = "2026-05-19T10:09:48.361Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/44/53/4f6bddbcde3c71e56d0aa1337ec95950f3d27dd4153e25aadf0feac71751/jiter-0.15.0-cp312-cp312-macosx_10_12_x86_64.whl", hash = "sha256:0e90a1c315a0226ec822d973817967f9223b7701546c8c2a7913e7ab0926294d", size = 308793, upload-time = "2026-05-19T10:07:35.25Z" }, - { url = "https://files.pythonhosted.org/packages/01/84/c01099b59a285a1ebba64ae93f62bfa036675340fd1b0045ae65890a0442/jiter-0.15.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:8c9004af7c8d67cce7f1aae1026fb55607f4aa600710d08ede3a3ce4aeefe7e0", size = 309570, upload-time = "2026-05-19T10:07:36.919Z" }, - { url = "https://files.pythonhosted.org/packages/58/64/8fb7f9d45bb98190355454cd04dad8d8f27223d6bd52f83af07f637168a6/jiter-0.15.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:c210f8b35dc6f30aafd4b4365ca89b9d1189f21ab49b8e68fa6322a847aef138", size = 336783, upload-time = "2026-05-19T10:07:38.694Z" }, - { url = "https://files.pythonhosted.org/packages/c3/b6/f5739011d009b3a30f6a53c5240979030ba29ae46a8c67e3a15759f7c37d/jiter-0.15.0-cp312-cp312-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:5f30bae8bc1c2d613e28e5af3e8cceb09b742f1c8a8a5f839fb67afaffc03b61", size = 363555, upload-time = "2026-05-19T10:07:40.832Z" }, - { url = "https://files.pythonhosted.org/packages/e5/12/98a9d9f766665e8a3b6252454e17cb0c464606a28cf2fa09399b003345fa/jiter-0.15.0-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:c60e71b6d10cfc284c9bf36bd885e8d44c46f688ce50aa91b5edd90181dea687", size = 452255, upload-time = "2026-05-19T10:07:42.62Z" }, - { url = "https://files.pythonhosted.org/packages/e8/d5/60f972840f79c5e7544fce567c56f1e4e50468f996baba3e78d823dd62a6/jiter-0.15.0-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:0ab068bce62a45aa3e7367eceaffb5dde60b7eb853be8dece45132e3d0ff4879", size = 373559, upload-time = "2026-05-19T10:07:44.201Z" }, - { url = "https://files.pythonhosted.org/packages/ee/cf/d46ef1234ba335aabc2f013210db8e0821a22f5e644a2e9449df199ecc23/jiter-0.15.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:fa248c9eb220197d363f688818dac2fd4b2f0cd7d843ca7105d652034823427d", size = 346055, upload-time = "2026-05-19T10:07:46.005Z" }, - { url = "https://files.pythonhosted.org/packages/f0/63/4d2749d8d54d230bad9b3a6b0d00cc28c6ff6b2fdffc26a8ccf76cc5a974/jiter-0.15.0-cp312-cp312-manylinux_2_31_riscv64.whl", hash = "sha256:2a77aadd57cac1682e4401a72724d2796d89a4ba129b1a5812aa94ee480826eb", size = 351406, upload-time = "2026-05-19T10:07:47.855Z" }, - { url = "https://files.pythonhosted.org/packages/d9/b9/9965b990035d8773328e0a8c8b457a87bf2b19f6c4126d9d99296be5d16a/jiter-0.15.0-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:2ae901f3a55bfafdde31d289590fa25e3245735a2b1e8c7cc15871710a002871", size = 389357, upload-time = "2026-05-19T10:07:49.665Z" }, - { url = "https://files.pythonhosted.org/packages/2d/55/9ddf903deda1413e87fed792f416b7123daee5b8efbad6a202a7421c36a5/jiter-0.15.0-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:f0b271b462769543716f92d3a4f90527df6ef5ed05ee95ec4137f513e21e1b77", size = 517263, upload-time = "2026-05-19T10:07:51.537Z" }, - { url = "https://files.pythonhosted.org/packages/e8/76/a0c40ad064d3a20a4fde231e35d56e9a01ce82164278180e82d5daf85469/jiter-0.15.0-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:2fb6a5d26af81fc0f00f9360a891e05cf755e149bba391c4d563adc54812973d", size = 548646, upload-time = "2026-05-19T10:07:53.196Z" }, - { url = "https://files.pythonhosted.org/packages/23/4f/eca9b954942916ba2f453891b8593ab444cd872396fe66a3936616f236f3/jiter-0.15.0-cp312-cp312-win32.whl", hash = "sha256:c2f6bb8b5216ab9e7873bc08b5d7bef2b8abbb578a3069bf1cd14a45d71d771d", size = 206427, upload-time = "2026-05-19T10:07:55.307Z" }, - { url = "https://files.pythonhosted.org/packages/95/bf/8ead82a87495149542748e828d153fd232a512a22c83b02c4815c1a9c7d8/jiter-0.15.0-cp312-cp312-win_amd64.whl", hash = "sha256:40b2c7e92c44a84d748d21706c68dc6ff8161d80b59c99d774721a0d2317d7c7", size = 197300, upload-time = "2026-05-19T10:07:56.651Z" }, - { url = "https://files.pythonhosted.org/packages/f4/e4/9b8a78fb2d894471bc344e37f1949bdd784bd914d031dba0ba3a40c71dd7/jiter-0.15.0-cp312-cp312-win_arm64.whl", hash = "sha256:cc0bc345cf2df9d1c00ac443f50d543c1ccfa8b0422cb85b1ab70d681c0b255b", size = 192702, upload-time = "2026-05-19T10:07:58.307Z" }, - { url = "https://files.pythonhosted.org/packages/73/38/505941b2b092fd5bbbd60a52a880db1173f1690ae6751bed3af1c9ddcb4e/jiter-0.15.0-graalpy312-graalpy250_312_native-macosx_10_12_x86_64.whl", hash = "sha256:631f13a3d04e97d4e083993b10f4b99530e3a10d953e2eb5e196b7dc7f812ce0", size = 303769, upload-time = "2026-05-19T10:09:42.203Z" }, - { url = "https://files.pythonhosted.org/packages/e7/95/a06692b29e77473f286e1ec1f426d3ca44d7b5843be8ad21d7a5f3fcdcc0/jiter-0.15.0-graalpy312-graalpy250_312_native-macosx_11_0_arm64.whl", hash = "sha256:b6c0ffae686c39bf3737be60793783267628783ea42545632c10b291105aee45", size = 305128, upload-time = "2026-05-19T10:09:43.657Z" }, - { url = "https://files.pythonhosted.org/packages/23/85/7270d7ad41d6061a25b950c6bf91d638bd9aacb113200a8c8d57a055fd67/jiter-0.15.0-graalpy312-graalpy250_312_native-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:1d54fb5b31dea401a41af3f8a7d2512e9b6a6a005491e6166c7e4ffab9639a9c", size = 340459, upload-time = "2026-05-19T10:09:45.452Z" }, - { url = "https://files.pythonhosted.org/packages/c8/8d/302cb2057b7513327b4d575cff6b1d066ee6431a5357fc3f8867cd684406/jiter-0.15.0-graalpy312-graalpy250_312_native-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:54d5d6090cdc1b7c9e780dfb04949a990adb1e301a2fc0bbcee7de4638d33f9a", size = 344469, upload-time = "2026-05-19T10:09:46.864Z" }, -] - [[package]] name = "llvmlite" version = "0.46.0" @@ -390,25 +364,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/78/a6/aae5cc2ca78c45e64b9ef22f089141d661516856cf7c8a54ba434576900d/numpy-2.3.5-cp312-cp312-win_arm64.whl", hash = "sha256:f28620fe26bee16243be2b7b874da327312240a7cdc38b769a697578d2100013", size = 10194667, upload-time = "2025-11-16T22:50:16.16Z" }, ] -[[package]] -name = "openai" -version = "2.38.0" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "anyio" }, - { name = "distro" }, - { name = "httpx" }, - { name = "jiter" }, - { name = "pydantic" }, - { name = "sniffio" }, - { name = "tqdm" }, - { name = "typing-extensions" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/8f/12/cfa322c5f5dd8fa21aab9a7a8e979e7a11123800f86ca8d82eb68a83d213/openai-2.38.0.tar.gz", hash = "sha256:798694c6cf74145541fda94325b6f8f72d8e1fd0262cc137c8d728177a6a4ce3", size = 772764, upload-time = "2026-05-21T21:23:42.105Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/0a/bf/ccff9be562e24207716d04ef9dc931c76aff0c89a7265da43e2104d7fe06/openai-2.38.0-py3-none-any.whl", hash = "sha256:ec6661c57b2dcc47414a767e6e3335c7ed3d19c9696999283a3c82e95c756a3c", size = 1344910, upload-time = "2026-05-21T21:23:39.636Z" }, -] - [[package]] name = "packaging" version = "26.2" @@ -580,30 +535,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/8f/cb/769cfc37177252872a45a71f3fbdde9d51b471a3f3c14bfe95dde3407386/python_multipart-0.0.29-py3-none-any.whl", hash = "sha256:2ddcc971cef266225f54f552d8fa10bcfbb1f14446caec199060daac59ff2d69", size = 29640, upload-time = "2026-05-17T17:29:45.69Z" }, ] -[[package]] -name = "regex" -version = "2026.5.9" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/dc/0e/49aee608ad09480e7fd276898c99ec6192985fa331abe4eb3a986094490b/regex-2026.5.9.tar.gz", hash = "sha256:a8234aa23ec39894bfe4a3f1b85616a7032481964a13ac6fc9f10de4f6fca270", size = 416074, upload-time = "2026-05-09T23:15:19.37Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/50/9b/6550044bc44e17c84d312c031c2ec42fbdb6a4ec4e29093be3a172d08772/regex-2026.5.9-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:57eeeb05db7979413dec5438f2db21d7ecbba787cde7a711df1a6f6df672aa06", size = 490451, upload-time = "2026-05-09T23:12:34.72Z" }, - { url = "https://files.pythonhosted.org/packages/1e/95/fc7ba4303b5a0f92446a12ee6778ef2c6c799233f5060042a31bf390cfe9/regex-2026.5.9-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:398c521292f4c7fb807001dcd54694d3a1fcafc179a36ad9cc56f98df85930b6", size = 292112, upload-time = "2026-05-09T23:12:36.285Z" }, - { url = "https://files.pythonhosted.org/packages/54/4b/ee27938d1b2c443e89a9a10e00d2d19aa5ee300cd3d61140644e93bb083e/regex-2026.5.9-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:f7a7c26137296beba7784de6eba69c6a93a63ccebc385e4962fe67e267a91225", size = 289599, upload-time = "2026-05-09T23:12:38.089Z" }, - { url = "https://files.pythonhosted.org/packages/d8/dd/ba103dc19614e25f3880800ca67ce093d6e21b325d72b8383c7bf906e9fa/regex-2026.5.9-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6441cc660d76107934a09c22167200839a0e89604a6297f78a974e66e931d2c0", size = 796732, upload-time = "2026-05-09T23:12:40.062Z" }, - { url = "https://files.pythonhosted.org/packages/cf/e7/f035b4fd858b050b0080bf302968dc0f59ba34e391872d54936758e6844e/regex-2026.5.9-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:91328f1c23d47595ca3ef0a7557fa129c5a23404b775c770697d2f35b33e0107", size = 865440, upload-time = "2026-05-09T23:12:42.059Z" }, - { url = "https://files.pythonhosted.org/packages/0a/51/8cd301ecc899aea28124357f729f4272f44de7806fc7ca02490bfbe253e8/regex-2026.5.9-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:93a7860539414dddaefba2b40f8771765ae17949d4c7182b876ce429e11a8309", size = 912329, upload-time = "2026-05-09T23:12:44.373Z" }, - { url = "https://files.pythonhosted.org/packages/cc/1e/3fbe2fa1e8cebd62f3bb7d3321cff1640aca2e240b51d9bd624aad949260/regex-2026.5.9-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:dd2810d22146b6d838acc5ec15602cb6b47920aa4e33015df3868eedfd20bab8", size = 801239, upload-time = "2026-05-09T23:12:46.268Z" }, - { url = "https://files.pythonhosted.org/packages/17/2f/6f6008682bf2cf98040a0d3153a8e557b6ab728d7713d045cee4ce544ab8/regex-2026.5.9-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:daff2bdbaf1d23e52fdff7c0b7bc2048b68f978df6a4d107ac981f94caef2e66", size = 777054, upload-time = "2026-05-09T23:12:48.051Z" }, - { url = "https://files.pythonhosted.org/packages/19/2b/eee0d20a6842ba04df4b8847a920b57ef56853f14ef85405473e586b605a/regex-2026.5.9-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:4eeb011098fcb77af513dcef521a3dbecbf8849b1e38940759d293b7a93f5026", size = 785098, upload-time = "2026-05-09T23:12:49.851Z" }, - { url = "https://files.pythonhosted.org/packages/4a/98/6fc1e6410feefb92159edaed5041992bfe390e8d26c721865434acbca558/regex-2026.5.9-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:ea9c8ecfa1b73c73b626534d6626e5340d429630943672b8480724f44e84b962", size = 860095, upload-time = "2026-05-09T23:12:51.666Z" }, - { url = "https://files.pythonhosted.org/packages/18/a3/bd855e0f2cb1a978ecf6fa6bb69632dd9c3f6ea3b81cde62fde14c9daec7/regex-2026.5.9-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:cd2846168eb9ee3c513902bc8225409cb1caab31d04728b145171fa1625d9621", size = 765762, upload-time = "2026-05-09T23:12:53.413Z" }, - { url = "https://files.pythonhosted.org/packages/dc/66/0ae8c092e60b14c79d24f8e0b7f0aea5bfbffdcab00b5483d13404d3c3a5/regex-2026.5.9-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:39617fb0cde9c0e6306dc70e3bfc096f3da793219879f7ae7aa341a69fbdcf6d", size = 852100, upload-time = "2026-05-09T23:12:55.256Z" }, - { url = "https://files.pythonhosted.org/packages/21/de/8dfde60fc1b21c946a893ba273403b72617edb261370cb1087099a83f088/regex-2026.5.9-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:fd03c4f0e33280d15cae17159b899245d6b7c53d21def19b263b39655061f5ce", size = 789479, upload-time = "2026-05-09T23:12:57.573Z" }, - { url = "https://files.pythonhosted.org/packages/c3/1c/bdcc98f9a4af4fdd166c74941174619ccff4726d3ce32faa8e9a2ecd38dd/regex-2026.5.9-cp312-cp312-win32.whl", hash = "sha256:164eba9b755ea6f244b0d881196fbc1fac09714e9782c9e2732b813142033c8e", size = 266699, upload-time = "2026-05-09T23:12:59.14Z" }, - { url = "https://files.pythonhosted.org/packages/78/87/240d36864f9e48ace85f72e79ced97ceb7f27ce87739a947dcb834b4e6bc/regex-2026.5.9-cp312-cp312-win_amd64.whl", hash = "sha256:86f40a5d6444db30a125c9c9177e6b25dad981cbc37451fd838f145e6edac92e", size = 277783, upload-time = "2026-05-09T23:13:00.789Z" }, - { url = "https://files.pythonhosted.org/packages/4f/b5/7b30f312b0669dff5beebe5b0989dc2d1a312b1a44fab852199c387a5b96/regex-2026.5.9-cp312-cp312-win_arm64.whl", hash = "sha256:96f5f58b54a063d7ea9dca08e1cf57bfe10499c4d579ee672da284f57f5f0070", size = 270513, upload-time = "2026-05-09T23:13:02.426Z" }, -] - [[package]] name = "requests" version = "2.34.2" @@ -650,20 +581,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/ec/e1/b2df4bc09a1e51ff664c1e17018a4274b42e5e9352e4a478ea540512dc88/starlette-1.0.1-py3-none-any.whl", hash = "sha256:7c0e69b2ee1c848bd54669d908500117a3ee13de603a21427e5c6fc1adf98dcd", size = 72802, upload-time = "2026-05-21T21:58:56.551Z" }, ] -[[package]] -name = "tavily-python" -version = "0.7.24" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "httpx" }, - { name = "requests" }, - { name = "tiktoken" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/92/d3/a6a9c24bfafed30b4ce3c3d685ab00806ad631c9742441f2597ec91f0002/tavily_python-0.7.24.tar.gz", hash = "sha256:6c8954193c6472231e813fe50cbd07806bd86c7228957675eb45875a44d58296", size = 27311, upload-time = "2026-04-27T17:26:50.511Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/63/ce/37e3aba0f359f540bfc57eb178f73d521161761f21e0aa28749f42750b11/tavily_python-0.7.24-py3-none-any.whl", hash = "sha256:1a750108de42c4b0b46e4c1b7b64aeaf7fad7d7bac9167927edce0081fe166c9", size = 20022, upload-time = "2026-04-27T17:26:48.885Z" }, -] - [[package]] name = "tenacity" version = "9.1.4" @@ -673,37 +590,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/d7/c1/eb8f9debc45d3b7918a32ab756658a0904732f75e555402972246b0b8e71/tenacity-9.1.4-py3-none-any.whl", hash = "sha256:6095a360c919085f28c6527de529e76a06ad89b23659fa881ae0649b867a9d55", size = 28926, upload-time = "2026-02-07T10:45:32.24Z" }, ] -[[package]] -name = "tiktoken" -version = "0.13.0" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "regex" }, - { name = "requests" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/e4/e5/5f3cb2159769d0f4324c0e9e87f9de3c4b1cd45848a96b2eb3566ad5ca77/tiktoken-0.13.0.tar.gz", hash = "sha256:c9435714c3a84c2319499de9a300c0e604449dd0799ff246458b3bb6a7f433c1", size = 38986, upload-time = "2026-05-15T04:51:27.153Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/85/8e/144bde4e01df66b34bb865557c7cd754ed08b036217ebd79c9db5e9048a9/tiktoken-0.13.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:32ac870a806cfb260a02d0cb70426aef02e038297f8ad50df5040bb5af360791", size = 1034888, upload-time = "2026-05-15T04:50:31.579Z" }, - { url = "https://files.pythonhosted.org/packages/36/18/d4ac9d20956cdebca04841316660ed584c2fecdc2b81722a28bc7ad3b1e4/tiktoken-0.13.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:4d9980f11429ed2d737c463bb1fb78cf330caa026adf002f714aced7849a687b", size = 982970, upload-time = "2026-05-15T04:50:32.961Z" }, - { url = "https://files.pythonhosted.org/packages/74/ed/6bb8d05b9f731f749fee5c6f5ca63e981143c826a5985877330507bd13b7/tiktoken-0.13.0-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:3f277ebea5edd7b8bf03c6f9431e1d67d517530115572b2dc1d465326e8f88c7", size = 1115741, upload-time = "2026-05-15T04:50:34.475Z" }, - { url = "https://files.pythonhosted.org/packages/34/de/2ca96b07a82d972b74fe4b46de055b79c904e45c7eab699354a0bfa697dc/tiktoken-0.13.0-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:a116178fa7e1b4065bff05214360373a65cac22f965be7b3f73d00a0dbfe7649", size = 1136523, upload-time = "2026-05-15T04:50:35.782Z" }, - { url = "https://files.pythonhosted.org/packages/ee/dc/9dafec002c2d4424378563cf4cf5c7fb93631d2a55013c8b87554ee4012c/tiktoken-0.13.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:2c397ddda233208345b01bd30f2fca79ff730e55731d0108a603f9bc57f6af3b", size = 1181954, upload-time = "2026-05-15T04:50:36.99Z" }, - { url = "https://files.pythonhosted.org/packages/a1/d0/1f8578c45b2f24759b46f0b50d31878c63c73e6bf0f2227e10ec5c5408dc/tiktoken-0.13.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:95097e4f89b06403976e498abf61a0ee73a7497e73fb599cb211d8197a054d91", size = 1240069, upload-time = "2026-05-15T04:50:38.221Z" }, - { url = "https://files.pythonhosted.org/packages/aa/90/28d7f154888610aa9237e541986beb62b479df29d193a5a0617dbb1514d0/tiktoken-0.13.0-cp312-cp312-win_amd64.whl", hash = "sha256:8f2d16e7a7c783ad81f36e457d046d1f1c8af70b22aec8a13238efe531977c41", size = 874748, upload-time = "2026-05-15T04:50:39.587Z" }, -] - -[[package]] -name = "tqdm" -version = "4.67.3" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "colorama", marker = "sys_platform == 'win32'" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/09/a9/6ba95a270c6f1fbcd8dac228323f2777d886cb206987444e4bce66338dd4/tqdm-4.67.3.tar.gz", hash = "sha256:7d825f03f89244ef73f1d4ce193cb1774a8179fd96f31d7e1dcde62092b960bb", size = 169598, upload-time = "2026-02-03T17:35:53.048Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/16/e1/3079a9ff9b8e11b846c6ac5c8b5bfb7ff225eee721825310c91b3b50304f/tqdm-4.67.3-py3-none-any.whl", hash = "sha256:ee1e4c0e59148062281c49d80b25b67771a127c85fc9676d3be5f243206826bf", size = 78374, upload-time = "2026-02-03T17:35:50.982Z" }, -] - [[package]] name = "typing-extensions" version = "4.15.0" @@ -867,10 +753,6 @@ dependencies = [ ] [package.optional-dependencies] -dev = [ - { name = "openai" }, - { name = "tavily-python" }, -] prod = [ { name = "google-genai" }, ] @@ -878,13 +760,11 @@ prod = [ [package.metadata] requires-dist = [ { name = "google-genai", marker = "extra == 'prod'", specifier = ">=0.3" }, - { name = "openai", marker = "extra == 'dev'", specifier = ">=1.40" }, { name = "pydantic", specifier = ">=2.7,<3.0" }, - { name = "tavily-python", marker = "extra == 'dev'", specifier = ">=0.5" }, { name = "zeroforce", editable = "packages/zeroforce" }, { name = "zeroforce-schemas", editable = "packages/schemas" }, ] -provides-extras = ["dev", "prod"] +provides-extras = ["prod"] [[package]] name = "zeroforce-schemas" From 0d777991fc4ed1734d25962f941426fa105e5047 Mon Sep 17 00:00:00 2001 From: justin06lee Date: Fri, 22 May 2026 16:26:57 -0700 Subject: [PATCH 12/18] feat(web): DraftGraph types + vision/seed API helpers --- apps/web/lib/api.ts | 17 ++++++++++++++--- apps/web/lib/types.ts | 16 ++++++++++++++++ 2 files changed, 30 insertions(+), 3 deletions(-) diff --git a/apps/web/lib/api.ts b/apps/web/lib/api.ts index ae8d414..30b6190 100644 --- a/apps/web/lib/api.ts +++ b/apps/web/lib/api.ts @@ -1,4 +1,4 @@ -import type { AnalysisSummary, JobRecord, SimulateResponse } from "./types"; +import type { AnalysisSummary, DraftGraph, JobRecord, SimulateResponse } from "./types"; async function jsonOrThrow(res: Response) { const body = await res.json().catch(() => ({})); @@ -9,15 +9,26 @@ async function jsonOrThrow(res: Response) { return body; } -export async function startAnalysis(input: string, mode = "company"): Promise<{ id: string }> { +export async function startAnalysis( + input: string, + mode = "company", + seedGraph?: DraftGraph, +): Promise<{ id: string }> { const res = await fetch("/api/v1/analyze", { method: "POST", headers: { "Content-Type": "application/json" }, - body: JSON.stringify({ input, mode, max_nodes: 16 }), + body: JSON.stringify({ input, mode, max_nodes: 16, seed_graph: seedGraph ?? null }), }); return jsonOrThrow(res); } +export async function uploadWhiteboard(file: File): Promise { + const form = new FormData(); + form.append("file", file); + const res = await fetch("/api/v1/vision", { method: "POST", body: form }); + return jsonOrThrow(res) as Promise; +} + export async function getAnalysis(id: string): Promise { return jsonOrThrow(await fetch(`/api/v1/analyses/${id}`)); } diff --git a/apps/web/lib/types.ts b/apps/web/lib/types.ts index 0c4d7a0..a7c796b 100644 --- a/apps/web/lib/types.ts +++ b/apps/web/lib/types.ts @@ -75,3 +75,19 @@ export interface SimulateResponse { p95_rounds: number; p99_rounds: number; } + +export interface DraftNode { + id: string; + label: string; +} + +export interface DraftEdge { + source: string; + target: string; +} + +export interface DraftGraph { + nodes: DraftNode[]; + edges: DraftEdge[]; + focal_node: string; +} From c78e1c63b6d3137bd6a16ba9112437e9fa7c741e Mon Sep 17 00:00:00 2001 From: justin06lee Date: Fri, 22 May 2026 16:28:41 -0700 Subject: [PATCH 13/18] feat(web): whiteboard photo upload entry on home page --- apps/web/app/page.tsx | 30 +++++++++++++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-) diff --git a/apps/web/app/page.tsx b/apps/web/app/page.tsx index 890ccb1..14f5197 100644 --- a/apps/web/app/page.tsx +++ b/apps/web/app/page.tsx @@ -3,7 +3,7 @@ import { useRouter } from "next/navigation"; import { useState } from "react"; import HistorySidebar from "@/components/HistorySidebar"; -import { startAnalysis } from "@/lib/api"; +import { startAnalysis, uploadWhiteboard } from "@/lib/api"; const EXAMPLES = [ "Apple semiconductor supply chain", @@ -30,6 +30,21 @@ export default function Home() { } } + async function onPhoto(e: React.ChangeEvent) { + const file = e.target.files?.[0]; + if (!file) return; + setLoading(true); + setError(null); + try { + const draft = await uploadWhiteboard(file); + sessionStorage.setItem("zeroforce.draft", JSON.stringify(draft)); + router.push("/confirm"); + } catch (err) { + setError((err as Error).message); + setLoading(false); + } + } + return (
@@ -139,8 +143,8 @@ export default function Confirm() { > {draft.nodes.map((n) => )} - - + + ))} From 01bfcce629996d61a1fe6a62e49ea31312349362 Mon Sep 17 00:00:00 2001 From: justin06lee Date: Fri, 22 May 2026 16:36:29 -0700 Subject: [PATCH 17/18] =?UTF-8?q?docs(infra):=20Gemini-only=20sweep=20?= =?UTF-8?q?=E2=80=94=20drop=20dev/Cerebras/Tavily,=20remove=20dead=20verte?= =?UTF-8?q?x=20provider?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.7 (1M context) --- .env.example | 20 ++++--------- README.md | 25 +++++++++-------- infra/docker-compose.dev.yml | 11 ++------ infra/docker/Dockerfile.backend | 2 +- infra/terraform/data.tf | 13 --------- .../providers/zeroforce_providers/fake.py | 3 +- .../providers/zeroforce_providers/lazy.py | 2 +- .../providers/zeroforce_providers/vertex.py | 28 ------------------- 8 files changed, 25 insertions(+), 79 deletions(-) delete mode 100644 packages/providers/zeroforce_providers/vertex.py diff --git a/.env.example b/.env.example index b792561..f5d49a2 100644 --- a/.env.example +++ b/.env.example @@ -1,24 +1,16 @@ # zeroforce environment. Copy to .env.local (gitignored) and fill in. -# Provider mode hinge (spec §6.6). One of: fake | dev | prod -# fake = deterministic offline providers (default; no keys needed) -# dev = Cerebras + Tavily -# prod = Gemini 3.5 Flash (Vertex) + Vertex grounding -ZEROFORCE_MODE=fake +# Provider mode hinge (spec §6.6). One of: prod | fake +# prod = Gemini 3.5 Flash (Vertex) + built-in Google Search grounding (default) +# fake = deterministic offline test double (no keys, no network; tests/CI + keyless demo) +ZEROFORCE_MODE=prod -# SQLite path (dev). Use :memory: for ephemeral. +# SQLite path (fake/local). Use :memory: for ephemeral. ZEROFORCE_DB=./data/zeroforce.db -# Optional single bearer token; if unset, auth is open (fake/dev convenience). +# Optional single bearer token; if unset, auth is open (local convenience). # ZEROFORCE_API_TOKEN= -# --- dev mode keys --- -# CEREBRAS_API_KEY= # from cloud.cerebras.ai -# CEREBRAS_MODEL=zai-glm-4.7 # extraction (accuracy-critical structured graph) -# CEREBRAS_TEXT_MODEL=gpt-oss-120b # report prose (cheaper/faster — frugal) -# CEREBRAS_BASE_URL=https://api.cerebras.ai/v1 # optional override -# TAVILY_API_KEY= - # --- prod mode (GCP Application Default Credentials must also be configured) --- # GCP_PROJECT= # GCP_REGION=us-central1 diff --git a/README.md b/README.md index c7b4780..8368101 100644 --- a/README.md +++ b/README.md @@ -30,7 +30,7 @@ engine that computes exact Expected Propagation Time (EPT) from every node. ## Quickstart ```bash -make install # uv workspace + provider SDKs (Cerebras/Tavily + Gemini) + frontend deps +make install # uv workspace + provider SDK (Gemini / google-genai) + frontend deps make dev # backend (:8080) + frontend (:3000) together; Ctrl-C kills both # open http://localhost:3000 ``` @@ -38,26 +38,27 @@ make dev # backend (:8080) + frontend (:3000) together; Ctrl-C kills bot `make dev` (`scripts/dev.sh`) frees the ports, then runs both in one foreground process group, so a single Ctrl-C tears down both. Run separately with `make backend` / `make web`. -Mode comes from `.env.local` (copy [`.env.example`](.env.example)). **`fake` needs no keys** and -runs the whole pipeline deterministically offline — good for a demo or development. +Mode comes from `.env.local` (copy [`.env.example`](.env.example)). The default is `prod` +(Gemini via GCP ADC). For a keyless offline demo set **`ZEROFORCE_MODE=fake`**, which runs the +whole pipeline deterministically with no keys and no network. -## Modes (the dev/prod hinge) +## Modes -One env var, `ZEROFORCE_MODE`: +One runtime — **Gemini** (via Vertex / GCP ADC) — with web grounding built into Gemini +(Google Search). One env var, `ZEROFORCE_MODE`: | Mode | LLM | Web grounding | Storage | Setup | |---|---|---|---|---| -| `fake` (default) | deterministic | deterministic | SQLite / memory | nothing | -| `dev` | **Cerebras** — `zai-glm-4.7` (extraction) + `gpt-oss-120b` (report) | Tavily | SQLite | `CEREBRAS_API_KEY`, `TAVILY_API_KEY` | -| `prod` | **Gemini 3.5 Flash** (Vertex) | built into Gemini (Google Search) | Firestore + Redis | GCP ADC | +| `prod` (default) | **Gemini 3.5 Flash** (Vertex) | built into Gemini (Google Search) | Firestore + Redis | GCP ADC | +| `fake` | deterministic | deterministic | SQLite / memory | nothing | -Model overrides: `CEREBRAS_MODEL`, `CEREBRAS_TEXT_MODEL`, `GEMINI_MODEL`. The split in `dev` is -frugal — the stronger model only on the accuracy-critical extraction, the cheaper one for prose. +Model override: `GEMINI_MODEL`. `fake` is an offline, deterministic test double — it needs +**no keys and no network** and exists only for tests/CI (and a quick keyless local demo). ### prod / GCP prod uses **Application Default Credentials** (no API key) and Gemini's **built-in** Google -Search grounding (Tavily is not used in prod). One-time auth: +Search grounding. One-time auth: ```bash make adc # = gcloud auth application-default login (or run setup_adc.sh) @@ -92,7 +93,7 @@ why that discipline is the project's most important safety mechanism. ``` packages/zeroforce RZF engine (built first; gating oracle suite) packages/schemas shared Pydantic API models -packages/providers LLM + search abstraction (fake/cerebras/tavily/gemini/vertex) + factory +packages/providers LLM + search abstraction (fake/gemini) + factory services/backend 6 service cores + FastAPI apps + orchestrator + storage + observability apps/web Next.js 14 dashboard infra/ Dockerfiles, docker-compose.dev, Terraform (Cloud Run, Firestore, …) diff --git a/infra/docker-compose.dev.yml b/infra/docker-compose.dev.yml index bfa347e..90941d3 100644 --- a/infra/docker-compose.dev.yml +++ b/infra/docker-compose.dev.yml @@ -1,17 +1,14 @@ # Local six-service stack (spec §5.4). Gateway embeds the orchestrator and fans out to the # four worker services over HTTP. Run from repo root: # docker compose -f infra/docker-compose.dev.yml up --build -# Provide CEREBRAS_API_KEY / TAVILY_API_KEY in the environment for ZEROFORCE_MODE=dev, or set -# ZEROFORCE_MODE=fake for fully offline operation. +# Runtime is Gemini (prod, via GCP ADC). Set ZEROFORCE_MODE=fake for fully offline operation. x-backend: &backend build: context: .. dockerfile: infra/docker/Dockerfile.backend environment: - ZEROFORCE_MODE: ${ZEROFORCE_MODE:-fake} - CEREBRAS_API_KEY: ${CEREBRAS_API_KEY:-} - TAVILY_API_KEY: ${TAVILY_API_KEY:-} + ZEROFORCE_MODE: ${ZEROFORCE_MODE:-prod} services: extractor: @@ -38,9 +35,7 @@ services: <<: *backend command: uvicorn zeroforce_backend.asgi:app --host 0.0.0.0 --port 8080 environment: - ZEROFORCE_MODE: ${ZEROFORCE_MODE:-fake} - CEREBRAS_API_KEY: ${CEREBRAS_API_KEY:-} - TAVILY_API_KEY: ${TAVILY_API_KEY:-} + ZEROFORCE_MODE: ${ZEROFORCE_MODE:-prod} ZEROFORCE_DB: /data/zeroforce.db ZEROFORCE_SERVICE_URLS: >- {"extractor":"http://extractor:8081","validator":"http://validator:8082", diff --git a/infra/docker/Dockerfile.backend b/infra/docker/Dockerfile.backend index d0e58ce..1e0a9b2 100644 --- a/infra/docker/Dockerfile.backend +++ b/infra/docker/Dockerfile.backend @@ -19,7 +19,7 @@ COPY data ./data RUN uv sync --all-packages --no-dev ENV PATH="/app/.venv/bin:$PATH" -ENV ZEROFORCE_MODE=dev +ENV ZEROFORCE_MODE=prod EXPOSE 8080 # Default: gateway (with embedded orchestrator). Override `command` for worker services. diff --git a/infra/terraform/data.tf b/infra/terraform/data.tf index 0c7c3e2..740592e 100644 --- a/infra/terraform/data.tf +++ b/infra/terraform/data.tf @@ -22,16 +22,3 @@ resource "google_storage_bucket" "uploads" { uniform_bucket_level_access = true force_destroy = false } - -# Provider keys for ZEROFORCE_MODE=dev fallbacks; prod uses Vertex via ADC and needs neither. -resource "google_secret_manager_secret" "cerebras" { - secret_id = "cerebras-api-key" - replication { auto {} } - depends_on = [google_project_service.apis] -} - -resource "google_secret_manager_secret" "tavily" { - secret_id = "tavily-api-key" - replication { auto {} } - depends_on = [google_project_service.apis] -} diff --git a/packages/providers/zeroforce_providers/fake.py b/packages/providers/zeroforce_providers/fake.py index 4cc77ae..b94202c 100644 --- a/packages/providers/zeroforce_providers/fake.py +++ b/packages/providers/zeroforce_providers/fake.py @@ -3,8 +3,7 @@ No network, no keys, fully reproducible. The fake LLM builds a valid supply-chain tree from any input so the whole pipeline runs end-to-end in tests and CI. The fake search returns deterministic snippets. These exist so nothing in the system requires live keys -to be exercised; real providers (cerebras/tavily/gemini) plug in via the factory when keys -are present. +to be exercised; the real provider (gemini) plugs in via the factory in prod. """ from __future__ import annotations diff --git a/packages/providers/zeroforce_providers/lazy.py b/packages/providers/zeroforce_providers/lazy.py index 877cc89..f5e82ad 100644 --- a/packages/providers/zeroforce_providers/lazy.py +++ b/packages/providers/zeroforce_providers/lazy.py @@ -1,6 +1,6 @@ """Lazy provider wrappers — build the real provider on first use, not at app startup. -Construction of a real provider (Gemini/Vertex client, Cerebras client) can fail (missing +Construction of a real provider (Gemini/Vertex client) can fail (missing ADC, missing key, network). Doing it eagerly at create_app() would crash the whole server — even endpoints that never touch an LLM (e.g. history). These wrappers defer construction so the server always starts; a construction error surfaces on the first call that needs the diff --git a/packages/providers/zeroforce_providers/vertex.py b/packages/providers/zeroforce_providers/vertex.py deleted file mode 100644 index 72f4ee2..0000000 --- a/packages/providers/zeroforce_providers/vertex.py +++ /dev/null @@ -1,28 +0,0 @@ -"""Prod search provider: Vertex grounding (spec §6.4). - -Grounding happens *inside* Gemini Pass A, so this issues no independent search query. It -exists only to expose Pass-A grounding metadata (cited URLs) to the citations panel. -""" - -from __future__ import annotations - -from .base import SearchResult - - -class VertexGroundingProvider: - name = "vertex-grounding" - - def __init__(self, gemini=None): - # Optionally linked to a GeminiProvider to surface its last_citations. - self._gemini = gemini - - async def search(self, query, max_results=5, include_domains=None, recency_days=None) -> list[SearchResult]: - # No independent search in prod; grounding is bundled into Gemini Pass A. - return [] - - @property - def citations(self) -> list[dict]: - return list(getattr(self._gemini, "last_citations", []) or []) - - async def fetch(self, url: str) -> str: # pragma: no cover - not used in prod path - raise NotImplementedError("Vertex grounding does not fetch URLs independently.") From 72f7206729b44b8ba11a466dcb8aee8ca19414a2 Mon Sep 17 00:00:00 2001 From: justin06lee Date: Fri, 22 May 2026 16:44:00 -0700 Subject: [PATCH 18/18] docs: align comments + ARCHITECTURE/RUNBOOK/PROGRESS with Gemini-only runtime Co-Authored-By: Claude Opus 4.7 (1M context) --- ARCHITECTURE.md | 13 +++++++------ PROGRESS.md | 3 +++ SECURITY.md | 2 +- docs/RUNBOOK.md | 18 ++++++++---------- .../backend/zeroforce_backend/gateway/app.py | 5 +++-- .../zeroforce_backend/service_client.py | 2 +- 6 files changed, 23 insertions(+), 20 deletions(-) diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index a30c431..d77e4b1 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -9,7 +9,8 @@ Full specification: [`.agents/project_specifications.md`](.agents/project_specif ## The three layers 1. **Extraction** — turn natural language into a weighted directed dependency graph. - Prod: Gemini 3.5 Flash two-pass (ground → structure). Dev: Tavily + Cerebras. Fake: deterministic. + Prod: Gemini 3.5 Flash two-pass (ground → structure; web grounding via Google Search is + built into Gemini). Fake: deterministic offline double for tests/CI. 2. **Computation** — the RZF engine computes exact Expected Propagation Time (EPT) per node via Markov-chain DP. This is `packages/zeroforce/` and was built first. 3. **Presentation** — Next.js 14 dashboard: force graph, impact ranking, cascade animation, @@ -62,12 +63,12 @@ Frontend ─▶ Gateway ─▶ Orchestrator ─▶ ServiceClient ─▶ {extract └────────────────────────────┘ (SQLite/Firestore jobs + analyses; Redis/dict extraction cache) ``` -## The dev/prod/fake hinge +## The prod/fake hinge One env var, `ZEROFORCE_MODE`: -- `fake` (default) — deterministic offline providers; no keys, no network. Powers tests/CI. -- `dev` — Cerebras (`CEREBRAS_API_KEY`) + Tavily (`TAVILY_API_KEY`). -- `prod` — Gemini 3.5 Flash (Vertex, ADC) + Vertex grounding. +- `prod` (default) — Gemini 3.5 Flash via Vertex (GCP ADC); web grounding is built into Gemini + via Google Search. +- `fake` — deterministic offline providers; no keys, no network. Powers tests/CI. Identical I/O contract across modes; only source-attribution metadata and call count differ. Application code imports only the `LLMProvider` / `SearchProvider` protocols, never a vendor SDK. @@ -77,7 +78,7 @@ Application code imports only the `LLMProvider` / `SearchProvider` protocols, ne ``` packages/zeroforce RZF engine (built first; gating oracle suite) packages/schemas shared Pydantic API models (engine types re-exported) -packages/providers LLM + search abstraction (fake/cerebras/tavily/gemini/vertex) + factory +packages/providers LLM + search abstraction (fake/gemini/vertex) + factory services/backend 6 service cores + FastAPI apps + orchestrator + storage + observability apps/web Next.js 14 dashboard infra/ Dockerfiles, docker-compose.dev, Terraform (Cloud Run, Firestore, …) diff --git a/PROGRESS.md b/PROGRESS.md index b3a493b..e67ea96 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -49,6 +49,9 @@ Started autonomous full build 2026-05-22. Operator asleep; no questions allowed. forcing 6 live HTTP processes for every test/CI run. - **Provider modes:** `ZEROFORCE_MODE = fake | dev | prod`. `fake` = deterministic offline (default for tests/CI). `dev` = NVIDIA+Tavily (needs keys). `prod` = Gemini+Vertex (needs GCP). + > UPDATE (post-build): the runtime was later collapsed to **Gemini-only**. The `dev` mode and + > the Cerebras/Tavily providers were removed; `ZEROFORCE_MODE` is now `prod` (Gemini via + > Vertex/ADC, default; web grounding built into Gemini) | `fake` (offline test double only). - **Storage abstraction:** `Repository` protocol; `SQLiteRepository` (default/dev), `FirestoreRepository` (prod, untested). Cache: `dict` (default) / `Redis` (prod). - **uv workspace** at repo root; members = `packages/*` + `services/*`. diff --git a/SECURITY.md b/SECURITY.md index 3d9ac7b..b5148af 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -14,7 +14,7 @@ Threat model and mitigations (spec §12), with implementation status in this rep ## Operational notes -- **No secrets are committed.** Verify with `git grep -nE "(Cerebras|TAVILY|API_KEY)=.+"` returning nothing. +- **No secrets are committed.** Verify with `git grep -nE "(API_KEY|SECRET|TOKEN)=.+"` returning nothing. - **Chaos**: a provider outage mid-pipeline fails the job cleanly with `llm_provider_error` (tested in `test_provider_outage_fails_job_gracefully`) rather than hanging or crashing. - **Before public deployment**: enable rate limiting, write Firestore security rules, add the diff --git a/docs/RUNBOOK.md b/docs/RUNBOOK.md index f10e6a0..8c9a506 100644 --- a/docs/RUNBOOK.md +++ b/docs/RUNBOOK.md @@ -11,10 +11,10 @@ cd apps/web && npm install && API_BASE=http://localhost:8080 npm run dev ## Local — six-service stack (docker-compose) ```bash -# offline: -docker compose -f infra/docker-compose.dev.yml up --build -# live dev providers: -ZEROFORCE_MODE=dev CEREBRAS_API_KEY=... TAVILY_API_KEY=... \ +# offline (tests/CI double): +ZEROFORCE_MODE=fake docker compose -f infra/docker-compose.dev.yml up --build +# live Gemini providers (needs GCP ADC): +ZEROFORCE_MODE=prod GCP_PROJECT=your-project \ docker compose -f infra/docker-compose.dev.yml up --build ``` @@ -25,9 +25,8 @@ Gateway on :8080 (fans out to extractor:8081, validator:8082, engine:8083, repor | Mode | LLM | Search | Storage | Keys | |---|---|---|---|---| +| `prod` (default) | Gemini 3.5 Flash (Vertex) | Google Search grounding (built into Gemini) | Firestore + Redis | GCP ADC, `GCP_PROJECT` | | `fake` | deterministic | deterministic | SQLite/memory | none | -| `dev` | Cerebras | Tavily | SQLite | `CEREBRAS_API_KEY`, `TAVILY_API_KEY` | -| `prod` | Gemini 3.5 Flash (Vertex) | Vertex grounding | Firestore + Redis | GCP ADC, `GCP_PROJECT` | ## Production deploy (Terraform → Cloud Run) — NOT YET APPLIED @@ -47,9 +46,8 @@ cd infra/terraform terraform init terraform apply -var "project_id=$PROJECT" -var "region=$REGION" -# 3. Load provider secrets (only needed if any service runs in dev mode) -echo -n "$CEREBRAS_API_KEY" | gcloud secrets versions add cerebras-api-key --data-file=- -echo -n "$TAVILY_API_KEY" | gcloud secrets versions add tavily-api-key --data-file=- +# 3. Provider auth: Gemini uses GCP Application Default Credentials (ADC) via the Cloud Run +# service account — no provider API keys to load. Ensure the SA has Vertex AI access. ``` Outputs `gateway_url` and `web_url`. Workers stay private (invoked only by the gateway SA). @@ -69,7 +67,7 @@ k6 run -e BASE=http://localhost:8080 tests/load/analyze_load.js ## Beta onboarding (Phase 5 — process, not code) 1. Collect the partner's focal firm + key tier-1 suppliers. -2. Run an analysis in `dev` mode; review the extracted graph for missing major suppliers +2. Run an analysis in `prod` mode; review the extracted graph for missing major suppliers (validator notes flag normalization issues). 3. Validate top-3 risk nodes with the partner's procurement team. 4. Iterate weights via the what-if panel; capture the resilience deltas. diff --git a/services/backend/zeroforce_backend/gateway/app.py b/services/backend/zeroforce_backend/gateway/app.py index 4e6c391..a1736c1 100644 --- a/services/backend/zeroforce_backend/gateway/app.py +++ b/services/backend/zeroforce_backend/gateway/app.py @@ -1,7 +1,8 @@ """Gateway FastAPI app — auth, routing, all /api/v1 endpoints (spec §9). -In dev/fake this single app embeds the pipeline via InProcessServiceClient. In prod/compose -it can be pointed at the per-service apps via HTTPServiceClient (set ZEROFORCE_SERVICE_URLS). +In the default (single-process) mode this app embeds the pipeline via InProcessServiceClient; +in split-service/compose deployments it can be pointed at the per-service apps via +HTTPServiceClient (set ZEROFORCE_SERVICE_URLS). """ from __future__ import annotations diff --git a/services/backend/zeroforce_backend/service_client.py b/services/backend/zeroforce_backend/service_client.py index 8104ffa..4f9a031 100644 --- a/services/backend/zeroforce_backend/service_client.py +++ b/services/backend/zeroforce_backend/service_client.py @@ -2,7 +2,7 @@ The orchestrator talks to the 5 worker services through this interface. Two impls: - InProcessServiceClient: calls each service's core directly. Default; offline + testable; - used in single-process dev and all tests. + used in single-process (in-process) mode and all tests. - HTTPServiceClient: calls each service's FastAPI app over HTTP. Used by docker-compose (six containers) and prod (one Cloud Run service each). """