Skip to content

Fall back to reasoning_content when model content is empty#245

Open
boskodev790 wants to merge 1 commit into
mini-router:mainfrom
boskodev790:sn74-boskodev790
Open

Fall back to reasoning_content when model content is empty#245
boskodev790 wants to merge 1 commit into
mini-router:mainfrom
boskodev790:sn74-boskodev790

Conversation

@boskodev790

Copy link
Copy Markdown

Problem

_parse_completion in src/trinity/llm/openai_compatible_pool.py builds the
completion text from message["content"] only. Reasoning models served over
OpenAI-compatible APIs — which is the entire live default pool in
configs/models.chutes.yaml (DeepSeek-V3.2-TEE, GLM-5-TEE, Kimi-K2.5-TEE)
— can return the answer in a separate reasoning_content channel while
content is null or "". This happens most often when the visible channel is
cut off by the token cap (finish_reason="length"), which is easy to hit with
reasoning_param: null and max_tokens: 4096.

When that happens res.text is "", so the grader extracts nothing and scores
the turn 0.0. Because this is the response-parse step shared by every call, it
would zero the single-model baselines and the router alike, not just one turn.

Note: whether a given empty-score eval is caused by this depends on the
provider's response shape — some reasoning models instead embed the chain of
thought in content as <think>…</think> and can be truncated by the token
cap, which is a separate concern. This PR fixes the specific case where the
answer is delivered in a separate reasoning_content field.

Fix

Fall back to reasoning_content when content is null/empty. A non-empty
content always wins, so ordinary completions are byte-for-byte unaffected —
this only rescues replies that would otherwise be dropped to "".

content = message.get("content")
if content is None or content == "":
    content = message.get("reasoning_content")

Testing

  • New regression tests in tests/test_pool_parse.py: null-content→fallback,
    empty-content→fallback, non-empty-content precedence, and
    null-without-reasoning stays "".
  • Full router suite green: 250 passed (was 246).
  • End-to-end (mock transport, no network): a reasoning reply with content:null
    and the answer in reasoning_content now flows through chat()
    score_text("math500", …) = 1.0, where before it was 0.0.

Scope / notes

  • Touches only the shared parser + its tests (53 lines, no refactor). Covers the
    chutes / fireworks / openrouter providers that share _parse_completion.
  • src/trinity/llm/minibridge_client.py has the same content-only pattern; it
    is not the default provider, so I left it out to keep this change scoped —
    happy to follow up if wanted.

Reasoning models served over OpenAI-compatible APIs (the live chutes pool:
DeepSeek/GLM/Kimi "thinking" variants) can return the answer in a separate
reasoning_content channel with content null or empty — most often when the
visible channel is truncated by the token cap (finish_reason="length").

_parse_completion read message.content only, so those replies produced
text="" and the grader scored them 0.0. Because this hits the response-parse
step shared by every call, it zeroes single-model baselines and the router
alike, not just one turn.

Fall back to reasoning_content when content is null/empty; a non-empty content
still always wins, so ordinary completions are unaffected. Adds regression
tests for the fallback, precedence, and the no-reasoning-content case.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant