Fall back to reasoning_content when model content is empty#245
Open
boskodev790 wants to merge 1 commit into
Open
Fall back to reasoning_content when model content is empty#245boskodev790 wants to merge 1 commit into
boskodev790 wants to merge 1 commit into
Conversation
Reasoning models served over OpenAI-compatible APIs (the live chutes pool: DeepSeek/GLM/Kimi "thinking" variants) can return the answer in a separate reasoning_content channel with content null or empty — most often when the visible channel is truncated by the token cap (finish_reason="length"). _parse_completion read message.content only, so those replies produced text="" and the grader scored them 0.0. Because this hits the response-parse step shared by every call, it zeroes single-model baselines and the router alike, not just one turn. Fall back to reasoning_content when content is null/empty; a non-empty content still always wins, so ordinary completions are unaffected. Adds regression tests for the fallback, precedence, and the no-reasoning-content case.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
_parse_completioninsrc/trinity/llm/openai_compatible_pool.pybuilds thecompletion text from
message["content"]only. Reasoning models served overOpenAI-compatible APIs — which is the entire live default pool in
configs/models.chutes.yaml(DeepSeek-V3.2-TEE,GLM-5-TEE,Kimi-K2.5-TEE)— can return the answer in a separate
reasoning_contentchannel whilecontentisnullor"". This happens most often when the visible channel iscut off by the token cap (
finish_reason="length"), which is easy to hit withreasoning_param: nullandmax_tokens: 4096.When that happens
res.textis"", so the grader extracts nothing and scoresthe turn
0.0. Because this is the response-parse step shared by every call, itwould zero the single-model baselines and the router alike, not just one turn.
Note: whether a given empty-score eval is caused by this depends on the
provider's response shape — some reasoning models instead embed the chain of
thought in
contentas<think>…</think>and can be truncated by the tokencap, which is a separate concern. This PR fixes the specific case where the
answer is delivered in a separate
reasoning_contentfield.Fix
Fall back to
reasoning_contentwhencontentisnull/empty. A non-emptycontentalways wins, so ordinary completions are byte-for-byte unaffected —this only rescues replies that would otherwise be dropped to
"".Testing
tests/test_pool_parse.py: null-content→fallback,empty-content→fallback, non-empty-content precedence, and
null-without-reasoning stays
"".content:nulland the answer in
reasoning_contentnow flows throughchat()→score_text("math500", …)= 1.0, where before it was0.0.Scope / notes
chutes / fireworks / openrouter providers that share
_parse_completion.src/trinity/llm/minibridge_client.pyhas the samecontent-only pattern; itis not the default provider, so I left it out to keep this change scoped —
happy to follow up if wanted.