[Bug] Custom-provider vision capability is ambiguous when model metadata is missing

[brunoflma]

Client or integration

Direct HTTP/API client through the OpenCodex proxy (/v1/responses, /v1/models).

Area

Proxy and routing · Catalog / models · Vision sidecar · Custom providers

Summary

This issue now tracks only the remaining TokenRouter case:

TR/moonshotai/kimi-k3-free

The previously reported OpenCode Zen vision cases were investigated and resolved by #1070. That PR classified the Zen models individually, including confirming that mimo-v2.5-free is natively vision-capable and that nemotron-3-ultra-free is text-only and should use the vision fallback.

The remaining failure is different because TR is a custom TokenRouter provider in the reporter configuration rather than a built-in OpenCodex registry provider. OpenCodex therefore has no canonical registry metadata from which to determine whether moonshotai/kimi-k3-free supports image input or requires the vision sidecar.

In the reporter's configuration, the model returns HTTP 200 but does not correctly identify a deterministic image, while another model on the same TokenRouter provider succeeds with the same request.

Reproduction

1. Configure the custom TR provider for TokenRouter.

2. Start OpenCodex.

3. Send a /v1/responses request using:

   TR/moonshotai/kimi-k3-free

4. Include a 16×16 opaque red PNG as the only input_image.

5. Ask the model to identify the dominant color.

6. Observe that the request returns HTTP 200 but does not correctly answer RED.

7. Repeat the same request with:

   TR/qwen/qwen3.8-max

8. Observe that the control model correctly processes the same image.

Version

Originally reproduced on @bitkyc08/opencodex 2.10.0.

The OpenCode Zen portion of the original report has since been resolved on dev by #1070. This issue remains open only for the custom TokenRouter route.

Operating system

Windows 11, using the Codex runtime through the local OpenCodex proxy.

The remaining behavior is expected to be provider/routing-specific rather than Windows-specific.

Provider and model

Custom provider:

TR → TokenRouter

Remaining affected model:

moonshotai/kimi-k3-free

Known working control on the same custom provider:

qwen/qwen3.8-max

Remaining problem

Custom-provider model rows do not necessarily carry authoritative modality metadata.

For a built-in provider, OpenCodex can maintain model-specific capability information such as:

• native image-input support;
• known text-only models;
• vision-sidecar requirements.

For a custom provider such as TR, OpenCodex cannot safely infer those properties from the provider name or model lineage alone.

The remaining question is therefore not whether the already-fixed Zen classifications should be expanded. It is how OpenCodex should handle missing or uncertain image capability metadata for custom-provider models without either:

• sending images directly to a text-only upstream, or
• unnecessarily replacing native image input with a sidecar for a model that actually supports vision.

Expected behavior

OpenCodex should have a deterministic policy for custom-provider models whose image-input capability is unknown.

A valid solution could include one or more of:

• preserving authoritative modality metadata returned by the custom provider's model catalog;
• allowing explicit per-model inputModalities / vision capability configuration;
• allowing explicit per-model vision-sidecar classification for custom providers;
• exposing unknown modality state clearly rather than treating unknown as proven vision support;
• ensuring the runtime and /v1/models expose the same effective capability decision.

The implementation must not globally classify models by name or vendor lineage when the actual custom endpoint may expose different capabilities.

Acceptance criteria

• TR/moonshotai/kimi-k3-free receives deterministic image handling based on actual or explicitly configured capability evidence.
• A custom-provider model with confirmed native vision support continues receiving the original image directly.
• A custom-provider model explicitly classified as text-only uses the vision fallback rather than receiving unsupported image content.
• Unknown capability is not silently converted into false capability certainty.
• /v1/models and request-time routing agree on the model's effective image capability.
• Regression coverage includes at least:
  • one custom-provider text-only model;
  • one custom-provider native-vision model;
  • one custom-provider model with unknown capability.

Resolved portions of the original report

• OpenCode Zen model classification: resolved by fix(zen): strip images for the Zen models that reject them (#1043) #1070.
• opencode-zen/mimo-v2.5-free: verified vision-capable.
• opencode-zen/nemotron-3-ultra-free: classified text-only.
• The broader Zen noVisionModels gap: resolved by fix(zen): strip images for the Zen models that reject them (#1043) #1070.

Those cases should not be considered acceptance criteria for this issue anymore.

Cross-references

• [Bug]: Vision sidecar never triggers for NVIDIA NIM text-only models (registry nvidia entry missing noVisionModels) #956 / stack 7/7: NIM vision classification, service repair, and the qwen3.8-max rename #980 — NVIDIA NIM modality classification work.
• fix(zen): strip images for the Zen models that reject them (#1043) #1070 — resolves the OpenCode Zen portion of this report.
• [Bug]: No management-plane support for provider headers — custom providers needing client fingerprints (Agent Router) 401 after headers are lost #959 / feat(providers): manage provider headers via PATCH and ocx provider edit --headers (#959) #1033 — custom-provider header management; related management surface but separate from modality capability.

Checks

• I searched existing issues and documentation.
• I removed secrets, tokens, account details, request credentials, and personal data.

[github-actions]

_{Automated translation bookkeeping — detected language: English.}

[github-actions]

Issue reopened

The report now contains the information required by the automated check. Thanks for updating it.

[brunoflma]

Client or integration

Codex runtime via the local OpenCodex proxy: /v1/responses for execution and /v1/models for catalog visibility.

Area

Proxy and routing · Catalog / models · Vision sidecar · Provider adapter

Summary

Following the NVIDIA-specific fix recorded in #956 (PR #980 merged into dev as 7343f0ba5914a96aa9eab0a1397bfa5454c286bc), a similar class of regression persists for other text-only or capability-uncertain models. These models do not advertise reliable modality coverage, yet they occasionally attempt to answer image requests without any sidecar guidance and therefore deliver either empty responses, hallucinated confidence, or explicit inability statements.

Four models in my environment are confirmed to produce HTTP 200 responses without recognizing the deterministic image:

Model	Failure mode
nvidia/nvidia/nemotron-3-ultra-550b-a55b	Explicit cannot-see answer after image payload
opencode-zen/mimo-v2.5-free	Empty response body
opencode-zen/nemotron-3-ultra-free	Same cannot-see pattern as NVIDIA
TR/moonshotai/kimi-k3-free	Empty response body

Controls on the same payload and agents answer "RED" correctly: anthropic/claude-sonnet-5, anthropic/claude-opus-5, DW/devworld/deepseek-v4-pro-r, TR/qwen/qwen3.8-max, google-antigravity/gemini-3.6-flash, plus both Agent Router transports AGR-OAI and AGR. The general request-build, authentication, and proxy routing paths work; the divergence is isolated to capability attribution and sidecar invocation for the emerging providers.

This sits alongside #956 rather than duplicating it: the NVIDIA path was repaired, but the corresponding decision surface now depends on per-provider hints that are not visibly synchronized with catalog state, registry confidence, or the exported configuration consumed by the Codex client.

Reproduction

1. Install @bitkyc08/opencodex 2.10.0 and start the proxy (ocx start).
2. Feed any flailing model this red 16×16 PNG via data:image/png;base64, inside a /v1/responses call and ask for the dominant color.
3. Observe the four failure cases above return readable or blank text with no vision sidecar acknowledgement.
4. Repeat against the controls above and note the correct RED response under the same conditions.

Version and runtime

• OpenCodex: 2.10.0.
• Local proxy confirmed healthy on port 10100 with upstreams reachable.
• Reference points to [Bug]: Vision sidecar never triggers for NVIDIA NIM text-only models (registry nvidia entry missing noVisionModels) #956 merged and tested on the dev branch — this issue therefore tracks what remains uncovered after that stack.

Evidence supporting root-cause hypothesis

• /v1/models rows for the four failing models show inputModalities: ["text"] or omit the field, so Codex grips only text-side affordances before any sidecar can intervene.
• Exported custom rows also lack inputModalities unless explicitly written, while the server-side registry keeps hint fields private to its import path — causing an off-by-one between runtime behavior and what users see in dashboards or clients.
• Logs during the failing probes contain no [vision]/sidecar traces, so no arbitration agent is invoked before the upstream rejection or hallucination.

Impact

• Failed models misbehave silently or confidently wrong without operator visibility.
• Automations that assume “image input supported” route around or ignore the sidecar but hit text-only ceilings.
• [Bug]: No management-plane support for provider headers — custom providers needing client fingerprints (Agent Router) 401 after headers are lost #959’s header work resolves transport for secrets, but does not address this capability-versus-catalog shadow state.

Suggested fix

1. Introduce a canonical per-model capability source of truth that merges registry hints, manual noVisionModels, and catalog export outputs, and make the runtime and export interfaces both read from it.
2. Extend persistent e2e coverage beyond NVIDIA: at least one text-only, one native-vision, and one unknown-provider probe per adapter, asserting non-empty, accurate answer or explicit 4xx.
3. Add an integration smoke test that walks /v1/models and /v1/responses for every known provider family in the local catalog build, so missing modality flags are caught before release.

Cross-references

• [Bug]: Vision sidecar never triggers for NVIDIA NIM text-only models (registry nvidia entry missing noVisionModels) #956 — NVIDIA/NIM vision classification fix (26 text-only, 16 vision-capable audited models).
• [Bug]: No management-plane support for provider headers — custom providers needing client fingerprints (Agent Router) 401 after headers are lost #959 — Provider headers management gap merged under feat: manage provider custom headers via PATCH and ocx provider edit --headers #961; excludes vision capability semantics discussed here.

Checks

• I searched existing issues and documentation.
• I removed secrets, tokens, headers, and prompt content.

[lidge-jun]

Sorry for the delay on your follow-up — it deserved a per-model answer rather than a general one, so here it is against what actually shipped in v2.10.2 (now current on npm).

Your four probes have three different dispositions:

opencode-zen/nemotron-3-ultra-free — fixed. It is in the text-only exception list now, so the proxy strips the image and the sidecar describes it instead of forwarding a part the endpoint rejects. That came from a live probe against opencode.ai/zen/v1 on 2026-08-05, recorded in devlog/_plan/260805_bug_fix_stack/002_zen_modality_probe.md. Please re-test on v2.10.2.

opencode-zen/mimo-v2.5-free — deliberately excluded, and I want to check your reading against ours. That same probe found mimo-v2.5-free accepts images, so it is intentionally absent from the list, with a negative assertion in tests/provider-registry-parity.test.ts to keep it that way. The reasoning: adding it would silently replace a real image with a caption, which is worse than a loud 400.

Your report describes it returning 200 while not recognizing the red-image probe — that is a different failure from rejecting the image. If the model accepts the part but answers as though it were blind, the honest classification is "accepts images, poor vision", and stripping the image would make our behavior worse, not better. If you disagree, the thing that would change my mind is a capture showing the raw request we sent and the model's response — specifically whether the image part was present on the wire.

nvidia/nvidia/nemotron-3-ultra-550b-a55b — the NVIDIA member was addressed by the #956 fix you cited; re-test on v2.10.2 and tell me if it still fails.

TR/moonshotai/kimi-k3-free — still blocked on you. TR (TokenRouter) is not a built-in registry provider, so its behavior comes entirely from your own provider config. Without that block, redacted, I cannot tell whether the models list, noVisionModels, or the input modality metadata is the gap. This is the one item where I genuinely cannot progress without you.

To be direct about the state: this issue is now partly fixed, partly a disagreement about classification, and partly waiting on one config paste. Keeping it open for the TR half. Your probe methodology — deterministic red image, text control first — is what made the Zen list possible, so thank you for that.