diff --git a/devlog/_plan/260809_vision_sidecar_model_filter/000_plan.md b/devlog/_plan/260809_vision_sidecar_model_filter/000_plan.md new file mode 100644 index 000000000..a1557b045 --- /dev/null +++ b/devlog/_plan/260809_vision_sidecar_model_filter/000_plan.md @@ -0,0 +1,154 @@ +# 260809 — vision sidecar model eligibility filter + +Base: `origin/dev@632743269`, worktree `/Users/jun/.codex/worktrees/34b7/opencodex`. +Cycle: docs-first. This unit writes the roadmap; no production code lands in this +work-phase. + +## Objective + +The dashboard's Vision sidecar model picker currently offers **every** model whose +provider is `openai` or `anthropic`, with no regard for whether that model can +actually accept an image. Users have been asking for more models, and the honest +answer has two halves: + +1. The picker is simultaneously **too wide** (it lists models that cannot see) and + **too narrow** (it hard-codes two provider names instead of asking about + capability). +2. The sidecar has exactly two executors — the OpenAI Responses forward path + (`src/vision/describe.ts`) and the Anthropic Messages path + (`src/vision/anthropic-describe.ts`). A model that is not reachable by one of + those two wire protocols cannot be a vision sidecar today no matter what the + picker shows. + +So the deliverable is a real **vision-capability filter** on both sides, plus a +guaranteed baseline entry per side, plus a dashboard card that shows the allowed +list in the compact delegation-panel form factor. + +## Constraints + +- The proxy is Bun-native TypeScript; `bun run typecheck` and `bun run test` gate + every layer. +- `src/vision/reasoning.ts` already owns effort normalization. The filter must not + duplicate or contradict `normalizeVisionReasoningForModel`. +- The GUI must not be the only gate: `PUT /api/sidecar-settings` has to reject an + ineligible vision model itself (`PLAN-BYPASS-NAMED-01` below). +- No change to routing, provider registry semantics, or the web-search sidecar's + behavior. Shared plumbing may be extracted, but web-search's option list keeps + its current contents in this unit. + +## The trap that shapes the whole design + +`applyProviderConfigHints` (`src/codex/catalog/provider-fetch.ts:574-582`) **adds** +`"image"` to a model's `inputModalities` when the model is listed in +`provider.noVisionModels`. That is deliberate: `noVisionModels` marks models the +PROXY describes images for, and the Codex app gates attachments client-side on +`input_modalities`, so a text-only entry would block the image before the sidecar +could ever run. + +The consequence for this unit is load-bearing: **catalog `inputModalities` is not a +truthful vision-capability signal.** A model that is in `noVisionModels` advertises +`["text","image"]` precisely *because* it is blind. Filtering on `inputModalities` +alone would let a blind model be chosen as the describer for other blind models. + +Therefore eligibility is a conjunction: + +``` +eligible(model) = advertisesImageInput(model) AND NOT isSidecarConsumer(model) +``` + +where `isSidecarConsumer` is `modelInList(provider.noVisionModels, id)` — the same +predicate `planVisionSidecar` uses to decide a model needs describing. + +## Work-phase map (dependency-ordered, PHASE-SPLIT-01) + +| # | Doc | Phase | Consumes | +|---|-----|-------|----------| +| 0 | this unit | roadmap | — | +| 1 | `010_vision_eligibility_core.md` | eligibility predicate + baselines in `src/vision/eligibility.ts` | catalog metadata accessors | +| 2 | `020_management_api_allowed_models.md` | `/api/sidecar-settings` exposes the allowed list and rejects ineligible writes | phase 1's predicate | +| 3 | `030_dashboard_vision_card.md` | dashboard card restyle + server-provided options | phase 2's payload | +| 4 | `040_stack_publication.md` | branch cascade, push, stacked PRs | phases 1-3 | + +Each phase closes with something independently verifiable: phase 1 with unit tests +over the predicate, phase 2 with route tests over both the 200 and the 400 path, +phase 3 with `lint:gui` + `build:gui` + a read-back screenshot, phase 4 with +`gh pr view` base refs. + +## Verifiers (PLAN-VERIFIER-REAL-01) + +Run before this plan was written, from the worktree root: + +| Command | Exit | Reads this unit's target? | +|---|---|---| +| `bun run typecheck` | 0 | yes — `tsconfig.json` compiles `src/**` and `gui/src/**`, which is where every phase writes | +| `bun run test` | 0 | yes — `tests/*.test.ts` is a flat glob over the whole directory, so a new `tests/vision-eligibility.test.ts` is picked up without configuration | +| `bun run lint:gui` | 0 | yes for phase 3 only — `gui/eslint.config.js` lints `gui/src/**`; it does **not** observe `src/**`, so it is not a gate for phases 1-2 | +| `bun run build:gui` | 0 | yes for phase 3 — Vite builds `gui/src` into `gui/dist` | +| `bun run privacy:scan` | 0 | partially — it scans the repo including `devlog/`, so it observes these documents, but it asserts nothing about the filter's behavior | + +`bun run lint:gui` does **not** observe `src/vision/*`; phase 1 and 2 acceptance +rows are covered by `typecheck` + `test`, not by lint. + +## Field chain (PLAN-FIELD-CHAIN-01) + +The unit adds one field to a wire payload — `visionModels` on the +`/api/sidecar-settings` GET response — and one derived value, the eligibility +boolean. Its chain: + +| Stage | Location | Note | +|---|---|---| +| creation | `visionEligibleModelOptions()` in `src/vision/eligibility.ts` (NEW) | derives from `listManagementModelRows` output plus config | +| serialization | `src/server/management/config-routes.ts` GET **and PUT** `/api/sidecar-settings` | both response bodies, so an optimistic update and a refetch cannot disagree | +| deserialization | `SidecarData` in `gui/src/pages/dashboard-shared.ts` | new optional `visionModels?: SidecarModelOption[]` — optional so a stale GUI against a new server, or a new GUI against a cached response, degrades to the old client-side list rather than rendering an empty picker | +| consumers | `use-dashboard-data.ts`: NEW `visionModels` memo + three field-by-field writes in `saveSidecar` (optimistic `next`, success `setSidecar`, session cache) and the hook's return object; `dashboard-overview-sections.tsx` (the vision `Select`) | the existing `sidecarModels` memo stays as-is and keeps serving web-search — `N/A` for it by design. The poll effect assigns `data.sidecar` wholesale, so it needs no edit; `CachedControls.sidecar` is typed `SidecarData`, so the optional field flows without a type change | +| validation | `PUT /api/sidecar-settings` in the same file, and the `visionSidecar` branch of `PUT /api/claude-code` in `agent-settings-routes.ts` | rejects a **provably blind** `vision.model` with 400; an unknown id is allowed | + +No enum gains a value in this unit, so the enum-consumer sweep is `N/A`. + +## Bypass (PLAN-BYPASS-NAMED-01) + +| Field | Value | +|---|---| +| tier | E4 — server-side request validation | +| executing surface | `PUT /api/sidecar-settings` in `src/server/management/config-routes.ts`, plus the Claude Code vision override in `src/server/management/agent-settings-routes.ts` | +| known bypass | editing `~/.opencodex/config.json` by hand and restarting; the config loader does not re-validate `visionSidecar.model` | +| residual risk | a hand-edited blind model stays configured and the sidecar produces useless descriptions; it fails at request time, not at write time | +| wording downgrade | yes, and deliberately. The gate rejects only models **positively known** to be unable to see. It is enforcement against a *proven-blind* selection and no barrier at all against an *unknown* one | +| final layer | none. `planVisionSidecar` stays permissive by design, so an operator can still point at a model the catalog has never heard of | + +**The gate and the picker are not the same set** (audit round 1, blocker 1). The +picker suggests; the gate forbids. Deriving one from the other would reject every +unknown id — including `custom-vision`, which +`tests/vision-reasoning-contract.test.ts:148-151` asserts must still save with +`providers: {}`. The rule is therefore: + +``` +picker option ⇐ eligible AND reachable by an executor AND known to some source +PUT rejection ⇐ modelAcceptsImageInput(...) === false (never on undefined) +``` + +## Scope boundary: the Claude Code override (audit round 1, blocker 5) + +Claude Code carries its own vision sidecar override +(`gui/src/pages/claude-code-sections.tsx:154-205`, persisted through +`PUT /api/claude-code`). Two halves, decided separately: + +- **Server: in scope.** The eligibility gate covers every route that sets a vision + describer, so `agent-settings-routes.ts` gets the same rejection. A second + unguarded write path would make the first gate decorative. +- **GUI: out of scope.** That surface is a freeform `` with a `` + of suggestions, not a constrained picker. Narrowing a deliberately freeform + field is a different product decision and is not smuggled into this unit. + +Requirement 2 ("both sides show only allowed models") is therefore read as: both +*backend families* (OpenAI and Anthropic) inside the dashboard vision picker — +which is the control the request was anchored on. + +## Out of scope + +- A third sidecar executor (e.g. a Gemini or xAI vision path). Adding a provider + family here would change routing surface, not just the picker, and belongs to its + own unit. +- Changing which models `noVisionModels` contains. +- Web-search sidecar option filtering. +- Merging the resulting PR stack. Publication is authorized; merging is not. diff --git a/devlog/_plan/260809_vision_sidecar_model_filter/001_capability_signal_inventory.md b/devlog/_plan/260809_vision_sidecar_model_filter/001_capability_signal_inventory.md new file mode 100644 index 000000000..454e9f9d9 --- /dev/null +++ b/devlog/_plan/260809_vision_sidecar_model_filter/001_capability_signal_inventory.md @@ -0,0 +1,103 @@ +# 001 — where a model's image capability actually comes from + +Research doc. No diffs here; the diffs live in the decade docs. + +## The four sources, in the order the runtime consults them + +1. **Native pinned metadata** — `src/codex/catalog/metadata.ts:117` + `nativeInputModalities(slug)` reads `src/codex/data/upstream-models.json`. + Verified contents for the seven supported native slugs: + + ``` + gpt-5.6-sol ["text","image"] + gpt-5.6-terra ["text","image"] + gpt-5.6-luna ["text","image"] + gpt-5.5 ["text","image"] + gpt-5.4 ["text","image"] + gpt-5.4-mini ["text","image"] + gpt-5.3-codex-spark (absent from snapshot → falls back to ["text","image"]) + ``` + + So **every** native OpenAI slug is image-capable. The native side of the picker + is not where over-listing happens. + +2. **Generated vendor metadata** — `src/generated/model-metadata.ts`, `DATA` rows + whose 4th column is a comma-joined modality string. `getModelMetadata(provider, + id)` and `getModelMetadataCaseInsensitive` return `input?: ("text"|"image"|"video")[]`. + Every `anthropic` row in that table carries `text,image`, including + `claude-haiku-4-5` and `claude-haiku-4-5-20251001`. + +3. **Live catalog rows** — `CatalogModel.inputModalities`, populated by + `catalogHintsFromModelsApiItem` (`provider-fetch.ts:938`) from the provider's + own `/models` payload, then post-processed by `applyProviderConfigHints`. + +4. **Operator config** — `provider.modelInputModalities[id]`, which + `configuredInputModalities` treats as the base before the `noVisionModels` + augmentation. + +## Live evidence from this machine + +`GET /api/models` (admin token, port 10100) returned 11 providers. The two the +picker currently sources from: + +``` +openai 7 rows, inputModalities: absent on every row +anthropic 11 rows, inputModalities: absent on every row +``` + +That absence is the second load-bearing fact. `listManagementModelRows` +(`src/server/management/model-rows.ts:45-54`) builds native rows by hand and never +attaches `inputModalities`; anthropic rows come through `dedupedRouted` from the +catalog, where the anthropic provider's `/models` response also omits it. + +**A naive `row.inputModalities?.includes("image")` filter would therefore empty the +picker completely.** Unknown must not be read as zero. The predicate has to fall +back to pinned/generated metadata before concluding a model cannot see, and when +all four sources are silent it must stay permissive. + +## The inverted signal + +`applyProviderConfigHints` (`provider-fetch.ts:574-582`) appends `"image"` to a +model listed in `provider.noVisionModels`. `tests/catalog-vision-sidecar-modalities.test.ts` +asserts exactly this: `glm-5.2`, a text-only model, comes out as +`["text","image"]`. + +`noVisionModels` is also the trigger for the sidecar itself — +`planVisionSidecar` (`src/vision/index.ts:237`) returns undefined unless +`modelInList(provider.noVisionModels, modelId)`. One list, two opposite meanings +depending on which side of the sidecar you stand on. + +Consequence for the predicate: membership in `noVisionModels` is a **hard +disqualifier** for being a describer, and it must be checked before the modality +list, because the modality list was rewritten by that very membership. + +## Which models a sidecar can actually reach + +`planVisionSidecar` has exactly two branches: + +- `backend === "anthropic"` → `describeImageAnthropic`, which requires an enabled + `adapter: "anthropic"`, `authMode: "oauth"` provider with a non-reauth active + account (`findAnthropicVisionProvider`, `src/vision/index.ts:171`). +- `backend === "openai"` → `describeImage` against + `${forwardProvider.baseUrl}/responses`, requiring a resolved OpenAI forward + sidecar (ChatGPT login). + +There is no third executor. A `xai/grok-4.5` row, image-capable though it is, +has no code path that would describe an image today. Eligibility must therefore be +scoped to the two backend families, not opened to every image-capable row in the +catalog — otherwise the picker would offer selections that silently produce no +plan at all. + +That is the honest answer to "can we allow other models?": **within the two +supported wire protocols, yes — and the current filter is the wrong shape. +Outside them, not without a new executor**, which this unit scopes out. + +## Baseline requirement + +The user requires `gpt-5.6-luna` to always appear when the GPT side is enabled and +a haiku model to always appear when Claude is registered. Both are image-capable by +the tables above, so the baselines are not exceptions to the capability rule — they +are a **presence** guarantee against an empty or unfetched catalog, which the live +evidence above shows is a real state (`/api/models` can be cold, and +`fetchAllModels` can return nothing while a provider is cooling down after a fetch +failure). diff --git a/devlog/_plan/260809_vision_sidecar_model_filter/002_audit_synthesis.md b/devlog/_plan/260809_vision_sidecar_model_filter/002_audit_synthesis.md new file mode 100644 index 000000000..3103e9ae3 --- /dev/null +++ b/devlog/_plan/260809_vision_sidecar_model_filter/002_audit_synthesis.md @@ -0,0 +1,138 @@ +# 002 — A-phase audit synthesis (round 1) + +Reviewer: independent subagent on `xai/grok-4.5`, high effort, read-only. +Verdict: `GO-WITH-FIXES (blockers=5)`. + +Every blocker below was **re-verified by the main agent against the tree** before +being accepted; none was taken on the reviewer's word. + +## Root-cause synthesis + +Blockers 1 and 2 are one defect, not two. I wrote the write-gate as *"reject what +is not in the picker"* while writing the bypass table as *"an operator may point +at a model the catalog does not know about"*. Those cannot both hold: an unknown +id is absent from the option list by construction, so the gate would reject +exactly the case the bypass table promises to allow. + +The correct rule follows from the tri-state the predicate already returns: + +``` +option list = eligible AND reachable AND known (a suggestion — may be narrow) +write gate = reject only when modelAcceptsImageInput(...) === false (a proof of harm) +``` + +`undefined` (nothing knows) belongs on the permissive side of the gate and the +conservative side of the list. Conflating "not suggested" with "forbidden" is the +error; the fix is to stop deriving the gate from the list. + +Blockers 3, 4, 6, 8 are all under-specification of a real call site — I named a +behavior and left the implementer to find the writes. Blocker 5 is a scope claim +I made too broadly. + +## Accept / rebut + +| # | Sev | Finding | Decision | Verification I ran | +|---|-----|---------|----------|--------------------| +| 1 | Critical | PUT 400 would reject unknown ids and break an existing contract test | **ACCEPT** | `tests/vision-reasoning-contract.test.ts:148-151` really does assert `putVision(custom, { model: "custom-vision" })` → 200 with `providers: {}`. My gate would have turned that red. | +| 2 | High | GET grandfathers the configured model, PUT does not | **ACCEPT** | Same root cause as 1; fixed by the same rule change. | +| 3 | High | `saveSidecar` drops `visionModels` on the success path | **ACCEPT** | `use-dashboard-data.ts:480-485`: `setSidecar({ webSearch: data.webSearch, vision: data.vision })` and the cache write both enumerate fields explicitly, so a new field is silently dropped. Three write sites, not one. | +| 4 | High | `config.providers.openai && !disabled` is not this repo's "OpenAI side enabled" | **ACCEPT** | `listOpenAiForwardSidecarCandidates` (`src/providers/openai-sidecar.ts:55-70`) additionally requires `isCanonicalOpenAiForwardProvider` (`openai-tiers.ts:32-36`): `openai-responses` + `forward` + canonical base URL. My predicate was strictly broader and asymmetric with the Anthropic side. | +| 5 | High | Requirement 2 ignores the Claude Code vision override | **ACCEPT IN PART** | `gui/src/pages/claude-code-sections.tsx:154-205` is a freeform `` with a ``, not a picker, and `agent-settings-routes.ts:1002-1014` only type-checks `model`. I accept the **server** half (the gate must cover every route that sets a vision describer) and rebut the **GUI** half (a freeform text input is deliberately freeform; narrowing its suggestions is a different product decision). Recorded as an explicit scope statement, not silence. | +| 6 | Medium | Card does not really adopt the delegation form factor | **ACCEPT** | `dash-delegation-summary` is applied to the *panel* at line 102; my draft put it on an inner row where `.dash-sidecar-card__row` already supplies the same flex rules. Redundant, and it would not match. | +| 7 | Medium | Wrong seam named for route tests | **ACCEPT** | The real seam is `handleManagementAPI` + `tests/helpers/management-auth`, demonstrated by `tests/vision-reasoning-contract.test.ts:12-31`. `model-routes.ts` documents a `deps.saveConfigPreservingClaudeCode` injection that the sidecar route does not use — it calls the bare import at line 469. | +| 8 | Medium | Compact CSS rule proposed in the wrong file | **ACCEPT** | The `min-width: clamp(10rem, 24vw, 11.5rem)` it must override lives in `styles-dashboard-workspace.css:104-110`, not `styles.css`. | +| 9 | Low | Stale line citations | **ACCEPT** | Re-pinned below. | +| 10 | Low | docs-site / CLI consumers omitted | **ACCEPT** | `src/cli/agent.ts` writes through the same PUT, so it inherits the corrected gate for free; `docs-site/src/content/docs/guides/sidecars.md` needs a sentence (SOT-SYNC-01, phase 2's C). | + +Reviewer claims I **rebut**: none outright. The one partial rebuttal (5) is scoped, +not dismissed. + +Reviewer claims I independently confirmed as *correct in my favor*: the +`:last-child` selector is robust (`Select` renders sibling `.custom-select` roots), +and the delegation panel itself needs no change because it already maps efforts to +raw values (`dashboard-overview-sections.tsx:119-122`). So comment 2's "여기처럼" +is a pattern reference, not a request to edit that panel. + +## Amendments applied + +1. `010` — `visionEligibleModelOptions` unchanged, but the doc now states the + list/gate asymmetry explicitly and adds a test asserting the tri-state. +2. `020` — the 400 guard is rewritten to fire only on + `modelAcceptsImageInput(...) === false`; `enabledVisionBackends` now uses + `listOpenAiForwardSidecarCandidates`; the same guard is applied to the Claude + Code vision override route; the seam is named correctly; a docs-site sync line + is added; two tests are added (unknown id keeps 200, Claude Code route rejects). +3. `030` — the card becomes `panel dash-delegation-summary`; all three + `visionModels` write sites are enumerated; the CSS rule moves to + `styles-dashboard-workspace.css`. +4. `000` — bypass table restated so the enforcement claim matches the code, and + the Claude Code scope boundary is written down. + +## Re-pinned anchors + +| Symbol | Real location | +|---|---| +| `planVisionSidecar` | `src/vision/index.ts:231` (its `modelInList` guard at 238) | +| `findAnthropicVisionProvider` | `src/vision/index.ts:172` | +| vision card markup | `gui/src/pages/dashboard-overview-sections.tsx:288-311` | +| delegation panel | `gui/src/pages/dashboard-overview-sections.tsx:102-104` | +| delegation raw effort labels | `gui/src/pages/dashboard-overview-sections.tsx:119-122` | +| sidecar select min-width | `gui/src/styles-dashboard-workspace.css:104-110` | +| `saveConfigPreservingClaudeCode` call in PUT | `src/server/management/config-routes.ts:469` | + +Confirmed accurate as originally written: `provider-fetch.ts:574-582`, +`config-routes.ts:380-393`, reasoning enum check `422-423`, `model-rows.ts:45-54`, +`use-dashboard-data.ts:454-462`. + +# Round 2 + +Same reviewer, re-audit of the amended documents. Verdict: +`GO-WITH-FIXES (blockers=1)`. Eight of nine prior findings confirmed **CLOSED**, +including the reviewer walking the `custom-vision` input through the amended +predicate step by step and reaching `undefined` (so the contract test stays +green), and confirming `.dash-delegation-controls .custom-select:last-child` +touches no other surface (`dash-delegation-controls` has exactly two users, and +the delegation panel's last child is a `