GPT-OSS-20B sometimes has nonsensical outputs that break expected openai-harmony format.

[EricApgar]

I think this is a GPT-OSS-20B model error. Like the LLM itself generates a bad output. The output is processed by the Harmony decoder which really just re-formats the tokens given an expected structure. It's expecting the message to come out of the LLM in a specific format that would let you separate the reasoning and the generated output. If the LLM bungles the generation, then the format is all off and the Harmony decoder breaks.

From what I see, it breaks because the LLM consistently outputs a whole bunch of "...". This happens pretty frequently - seems like a 20% chance per message. You could get around it by attempting to decode, and just re-generating the output if the decoding breaks, but 20% is super high. I don't want 20% of my messages to take twice as long because I have to regenerate them.

Snapshot

[EricApgar]

It's possible that this is caused by the Harmony format of messages on the front end and that if you removed that format, the LLM would be less likely to generate bad outputs. That would seem odd though because the whole point of the formatter is to structure prompts in a way that the LLM expects and was trained on so that it gives a better and more consistent result.

All I know is I haven't noticed this happening when I wasn't using the Harmony formatter.

Things to try?

• Mess with the settings in the actual generation command.
• Mess with other settings in the Harmony formatter before it makes it to generation.
• Look more online for solution.

[EricApgar]

I found this PR with a comment with someone experiencing my exact problem, but I don't really understand the resolution. I feel like the PR should have fixed all that and it didn't. I'll check that the version I'm running is actually the latest.

[EricApgar]

From what I can tell, the problem is that the LLM generates a whole mess of bad tokens before getting back on track to its normal responses. Why it generates 60 garbage tokens that are mostly "..."'s? I have no idea. How does it manage to generate this awful start and yet get back on track and finish with good content? I have no idea. Point is, the problem is easily patched by skipping over the bad tokens (which seem to consistently be at the start of the response) up until you encounter [..., '', 'analysis', ...] which seems to be what a normal message starts off with. This key phrase seems to be consistently still in the messages with a bad start, just later on in the sequence.

It doesn't solve the bad generation problem (which I still want to fix because every bad token generated is a waste of time), but it's easy to patch by chopping off all the bad tokens at the start.

Patch Fix

I patched this with this commit by finding and skipping over the bad tokens and now the harmony prompt formatter doesn't break.

[EricApgar]

I thought I patched this successfully, but I'm still getting problems with this occurring. I think the patch works sometimes, but if the entire sequence of generated tokens is trash, then it doesn't work.