docs(sglang): record the MTP penalty-accounting defect we filed but do not patch - #87
Draft
dorado269 wants to merge 1 commit into
Draft
docs(sglang): record the MTP penalty-accounting defect we filed but do not patch#87dorado269 wants to merge 1 commit into
dorado269 wants to merge 1 commit into
Conversation
…o not patch sglang commits accept_len tokens per decode step under MTP but feeds only output_ids[-1] to the sampling penalizers, so every other accepted token never enters penalty state. Filed as sglang#33643 (ours, DRAFT). Not carried as a local diff: cumulate_penalty_output_tokens runs only when penalizer_orchestrator.is_required, and nothing here sets a penalty. The gate that keeps it dead is the model, not us -- --sampling-defaults defaults to `model`, so a repetition_penalty in the served weights' generation_config would turn it on for every chat request. Upstream zai-org/GLM-5.2 carries none; our MXFP4 derivative was not on the host to confirm. Adds a "Filed upstream, not patched here" section for this class of row, since the page until now assumed every entry had a patch file to hold its argument. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
One row added to
deploy/docker/patch.upstream.status.mdfor an sglang defect I filed upstream as sglang#33643 (ours,dorado269, DRAFT) and deliberately did not patch locally.The defect: a decode step commits
accept_lentokens under MTP, butcumulate_penalty_output_tokensfeeds onlyoutput_ids[-1]to the sampling penalizers. Every other accepted token never enters penalty state and is never backfilled, sofrequency_penaltyends up effectively divided by the accepted length, andmin_new_tokenscounts decode steps instead of tokens — masking EOS to-inffor roughlyaccept_len× longer than the request asked.Why it needs a new section
The page opens with "Every patch under
deploy/docker/patches/", and every section is a patch directory. This row has no patch file, so there is no patch header to hold the argument the way## Maintenanceassumes. Rather than bend an existing table, this adds## Filed upstream, not patched herebefore## Not patches, and says in the intro and in## Maintenancethat the section runs the other way.The point of the row is that "we know, and here is why we are not carrying a diff" should be answerable from this page — otherwise the next person who finds #33643 re-argues it.
Why no local patch
The path is gated.
eagle_prepare_for_decodecallscumulate_penalty_output_tokens()only whenpenalizer_orchestrator.is_required, which needs some in-flight request to carry a non-defaultfrequency_penalty,presence_penalty,repetition_penaltyormin_new_tokens. Nothing in this repo sets one — grepped 2026-08-05, the onlypenaltyhits are unrelated prose about load and seek penalties.The non-obvious way it could turn on is the model rather than our configs:
--sampling-defaultsdefaults tomodel, so sglang copiesrepetition_penaltyout of the served weights'generation_config.jsoninto every chat request's defaults. Upstreamzai-org/GLM-5.2carries onlytemperatureandtop_pthere, read from the raw file rather than inferred from a search, so the gate stays shut. We serve the MXFP4 derivative and it was not on the host to check — that caveat is stated in the row rather than glossed.Verification
zai-org/GLM-5.2has norepetition_penaltydefaultgeneration_config.jsonfetched 2026-08-05--sampling-defaultsdefaults tomodelserver_args.pyin sglangmaingh issue view, 2026-08-05generation_config.jsonDocs only, no code or Dockerfile touched. Draft because #33643 is itself a draft; the
PR statecolumn should be refreshed if that changes before this merges.