Skip to content

fix(overflow): add hub GLM phrase to context-overflow detector - #314

Merged
linj-glitch merged 1 commit into
mainfrom
linj/hub-glm-overflow-phrase
Aug 5, 2026
Merged

fix(overflow): add hub GLM phrase to context-overflow detector#314
linj-glitch merged 1 commit into
mainfrom
linj/hub-glm-overflow-phrase

Conversation

@linj-glitch

@linj-glitch linj-glitch commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

The hub's GLM model returns "Input length N exceeds the maximum allowed input length of M tokens" with code "400" rather than "context_length_exceeded", so is_context_overflow returned false and the fallback in #302 was never consulted. Adds "exceeds the maximum allowed input length" to the OpenAI phrase list in both crates, with a test asserting the exact GLM error body shape.

Summary by CodeRabbit

  • Bug Fixes
    • Improved detection of input-length limit errors from OpenAI-compatible providers.
    • Added support for recognizing overflow errors returned through Hub and LiteLLM-wrapped GLM responses, including non-standard error codes.
    • These errors are now handled consistently as context overflow conditions.

Signed-off-by: Lin Jia <linj@nvidia.com>
@linj-glitch
linj-glitch requested a review from a team as a code owner August 5, 2026 20:03
@coderabbitai

coderabbitai Bot commented Aug 5, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 49250bdd-4f80-414d-aeb1-38e5d208f0b0

📥 Commits

Reviewing files that changed from the base of the PR and between 2a91472 and 67b6d8e.

📒 Files selected for processing (2)
  • crates/libsy-llm-client/src/backend.rs
  • crates/switchyard-components/src/backends/openai.rs

Walkthrough

Both OpenAI backend implementations recognize "exceeds the maximum allowed input length" as a context-overflow error. Regression tests cover LiteLLM-wrapped Hub GLM errors with structured code "400".

Changes

Context overflow detection

Layer / File(s) Summary
Overflow phrase and regression coverage
crates/libsy-llm-client/src/backend.rs, crates/switchyard-components/src/backends/openai.rs
Both OpenAI backends match the new overflow phrase. Tests verify detection for LiteLLM-wrapped Hub GLM errors with code "400".

Estimated code review effort: 1 (Trivial) | ~5 minutes

Poem

A rabbit hops through errors bright,
And finds the overflow phrase just right.
GLM’s code four-zero-zero
Now joins the test burrow.
Two backends guard the length tonight.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change to add hub GLM overflow handling.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

Comment @coderabbitai help to get the list of available commands.

@linj-glitch

Copy link
Copy Markdown
Contributor Author

Why phrase matching and not error code matching

A reviewer might ask why we're extending the phrase list rather than adding a structured code check, since the code path already short-circuits on error.code == "context_length_exceeded" before falling back to phrases.

The answer is that GLM (via the hub's LiteLLM proxy) doesn't emit a semantic error code. The actual response body is:

{"error": {"message": "Input length 877338 exceeds the maximum allowed input length of 639968 tokens", "code": "400"}}

code here is just the HTTP status stringified, not an OpenAI-style error code. There's nothing structured to match on — phrase detection is the only hook this provider gives us.

The existing comment in the source already covers the safety argument: a false positive on phrase matching triggers one bounded evict-and-retry, not an infinite loop, so erring toward matching the message is the right trade-off.

@linj-glitch
linj-glitch merged commit 65491db into main Aug 5, 2026
17 checks passed
@linj-glitch
linj-glitch deleted the linj/hub-glm-overflow-phrase branch August 5, 2026 21:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants