Skip to content

fix(profiles): gate DeepSeek thinking-off default on reasoning-hint compatibility #122

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
himorishige wants to merge 1 commit into
base: main
Choose a base branch
from
Open

fix(profiles): gate DeepSeek thinking-off default on reasoning-hint compatibility #122

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 11 additions & 2 deletions switchyard/lib/profiles/tier_target_builders.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,17 +20,26 @@
build_native_backend,
resolve_llm_target,
)
from switchyard.lib.processors.reasoning_hint import model_accepts_reasoning_hint
from switchyard.lib.profiles.deterministic_routing_config import (
DEFAULT_DETERMINISTIC_TIER_TIMEOUT_S,
)
from switchyard.lib.roles import LLMBackend


def apply_deepseek_overrides(target: LlmTarget) -> LlmTarget:
"""Apply benchmark-specific DeepSeek extras without clobbering callers."""
"""Apply benchmark-specific DeepSeek extras without clobbering callers.

The thinking-off default is a vLLM-side hint (``chat_template_kwargs``);
serving stacks on the :func:`model_accepts_reasoning_hint` deny list
reject the field outright (HTTP 400 ``Extra inputs are not permitted``),
so the default is gated on the same model-id check the LLM classifier
already uses for its own calls. The batch-priority header default is
provider-neutral and stays unconditional.
"""
default_body = (
{"chat_template_kwargs": {"enable_thinking": False}}
if "deepseek-v4" in target.model
if "deepseek-v4" in target.model and model_accepts_reasoning_hint(target.model)

@elyasmnvidian elyasmnvidian Jul 28, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice — gating the tier default on model_accepts_reasoning_hint matches what the classifier already does, so both paths now agree on when to send enable_thinking=false. (For context: apply_deepseek_overrides sets default request options for DeepSeek tier calls, and this hint is one of them.)

One thing to flag for later: both the tier default (this function) and the classifier default now depend on that one check being complete. The check works by looking for provider names inside the model id, so a new backend that rejects the hint will still receive it until someone remembers to add its name to _NO_REASONING_HINT_TAGS. That is easy to forget, and the failure shows up as a hard 400 in production.

Concrete suggestion: consider deciding this on the target/endpoint config rather than the model name. For example, add an optional accepts_reasoning_hint flag to the target (LlmTarget in llm_target.py), default it from model_accepts_reasoning_hint(model) when the config doesn't set it, and have both apply_deepseek_overrides and the classifier presets read that flag. Then a strict backend can be marked incompatible in its own config, with no code change and no name list to keep in sync.

@himorishige himorishige Jul 30, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review — agreed on both points.

On the fragility: yes, with this PR both the tier default and the classifier default hang off _NO_REASONING_HINT_TAGS, and a new strict backend silently inherits the hint until someone adds its name. Your Azure example on #123 already demonstrates a miss.

On moving the decision to the target config — one implementation note from reading current main: LlmTarget is owned by the Rust extension (switchyard_rust.components); llm_target.py only hosts coerce_llm_target() and helpers. So an accepts_reasoning_hint field on the target itself means a Rust-side schema change, not just a Python default. Two Python-side shapes that stay closer to the current layout:

  • extend llm_target_with_runtime_defaults(), which already maps model-name fragments (nemotron-3-super) to a thinking-off extra_body with an explicit-extra_body guard — so per-model request defaults keep living in one place; or
  • the per-provider settings map you sketched on fix(processors): treat Fireworks model ids as reasoning-hint incompatible #123. I posted measurements there suggesting the map earns its keep: for Fireworks the right move is to send a different knob (reasoning_effort: "none"), not merely to suppress this one, and a boolean flag can never express that.

Happy to keep this PR as the minimal consistency fix (both paths agreeing on one check) and follow up with the config/map work as a separate PR — your call on which shape you'd want.

else None
)
default_headers = (
Expand Down
15 changes: 15 additions & 0 deletions tests/test_deterministic_routing_profile.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -391,6 +391,21 @@ def test_caller_supplied_empty_body_wins(self) -> None:
out = apply_deepseek_overrides(target)
assert out.extra_body == {}

def test_hint_incompatible_model_id_skips_thinking_off(self) -> None:
"""Deny-listed serving stacks reject the vLLM hint with HTTP 400,
so the thinking-off default must not be injected for their ids.
The provider-neutral batch-priority header still applies."""
target = LlmTarget(
id="weak",
model="bedrock/deepseek-ai/deepseek-v4-flash",
format=BackendFormat.OPENAI,
api_key="k",
base_url="https://e/v1",
)
out = apply_deepseek_overrides(target)
assert not out.extra_body
assert out.extra_headers == {"X-Inference-Priority": "batch"}


class TestTierTimeoutDefaults:
"""Deterministic tiers get a bounded timeout unless callers set one."""
Expand Down