fix(overflow): add hub GLM phrase to context-overflow detector - #314
Conversation
Signed-off-by: Lin Jia <linj@nvidia.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (2)
WalkthroughBoth OpenAI backend implementations recognize ChangesContext overflow detection
Estimated code review effort: 1 (Trivial) | ~5 minutes Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Comment |
|
Why phrase matching and not error code matching A reviewer might ask why we're extending the phrase list rather than adding a structured code check, since the code path already short-circuits on The answer is that GLM (via the hub's LiteLLM proxy) doesn't emit a semantic error code. The actual response body is: {"error": {"message": "Input length 877338 exceeds the maximum allowed input length of 639968 tokens", "code": "400"}}
The existing comment in the source already covers the safety argument: a false positive on phrase matching triggers one bounded evict-and-retry, not an infinite loop, so erring toward matching the message is the right trade-off. |
The hub's GLM model returns "Input length N exceeds the maximum allowed input length of M tokens" with code "400" rather than "context_length_exceeded", so is_context_overflow returned false and the fallback in #302 was never consulted. Adds "exceeds the maximum allowed input length" to the OpenAI phrase list in both crates, with a test asserting the exact GLM error body shape.
Summary by CodeRabbit