vLLM issues with gemma4 tool calling

[mpetruc]

Hi,
i've been trying to use Late-cli with gemma--4-31b and vllm (0.23.0). I'm using the vllm-provided 'tool_chat_template_gemma4.jinja' and the required flags:

      --reasoning-parser gemma4
      --enable-auto-tool-choice
      --tool-call-parser gemma4

very frequently (but not 100% of the time), the loop stops after some tool_call, e.g.

<|tool_call>call:spawn_subagent{agent_type:<|"|>coder<|"|>,goal:<|"|>Act as an INSPECTOR subagent. Critically review the implementation of full SpeechRequest parameter support.

1. **Code Review**:
    [...]

2. **Audit**:
   [...]

3. **Verdict**: "[...].<|"|>}<tool_call|>

At this point i prompt "continue", but i'm not sure that the tool_call above actually got executed.

I also see blocks like this

<|channel>thought
<channel|>

but these seem to be benign.

Any idea why is this happening? is it vllm? the model itself (LilaRest/gemma-4-31B-it-NVFP4-turbo)? late? How to even tell?
thanks a lot!

[mpetruc]

this error just appeared for the first time in late, and it might provide more clues:

Error: stream error: API error (400): 3 validation errors:
  {'type': 'literal_error', 'loc': ('body', 3, 'ChatCompletionMessageFunctionToolCallParam', 'type'), 'msg': "Input should be 'function'", 'input': '', 'ctx': {'expected': "'function'"}}
  {'type': 'missing', 'loc': ('body', 3, 'ChatCompletionMessageCustomToolCallParam', 'custom'), 'msg': 'Field required', 'input': {'function': {'arguments': '{"command": "dir', 'name': ''}, 'id': '', 'index': 3, 'type': ''}}
  {'type': 'literal_error', 'loc': ('body', 3, 'ChatCompletionMessageCustomToolCallParam', 'type'), 'msg': "Input should be 'custom'", 'input': '', 'ctx': {'expected': "'custom'"}}

  File "/home/user1/vllm/vllm/entrypoints/serve/utils/api_utils.py", line 40, in create_chat_completion
    POST /v1/chat/completions [{'type': 'literal_error', 'loc': ('body', 3, 'ChatCompletionMessageFunctionToolCallParam', 'type'), 'msg': "Input should be 'function'", 'input': '', 'ctx': {'expected': "'function'"}}, {'type': 'missing', 'loc': ('body', 3, 'ChatCompletionMessageCustomToolCallParam', 'custom'), 'msg': 'Field required', 'input': {'function': {'arguments': '{"command": "dir', 'name': ''}, 'id': '', 'index': 3, 'type': ''}}, {'type': 'literal_error', 'loc': ('body', 3, 'ChatCompletionMessageCustomToolCallParam', 'type'), 'msg': "Input should be 'custom'", 'input': '', 'ctx': {'expected': "'custom'"}}]

vllm:

[WARN] error processing streaming response: no valid JSON data found in stream, path=/v1/chat/completions, 

Hopefully this is helpful.

[mlhher]

Hi, thank you for bringing this up. I mostly use llama.cpp so what I can give are only assumptions.
Llama.cpp had very similar behaviour up until a PR was merged: ggml-org/llama.cpp#18675

It would randomly stop streaming (mostly after tool calls) and crash when spawning subagents. Would be good to check if this happens more if the tool call is spawn subagent vs. any other tool call as you said not 100% of the times. Note that the crashing subagents fix also required ggml-org/llama.cpp#19924.

Error: stream error: API error (400): 3 validation errors:
  {'type': 'literal_error', 'loc': ('body', 3, 'ChatCompletionMessageFunctionToolCallParam', 'type'), 'msg': "Input should be 'function'", 'input': '', 'ctx': {'expected': "'function'"}}
  {'type': 'missing', 'loc': ('body', 3, 'ChatCompletionMessageCustomToolCallParam', 'custom'), 'msg': 'Field required', 'input': {'function': {'arguments': '{"command": "dir', 'name': ''}, 'id': '', 'index': 3, 'type': ''}}
  {'type': 'literal_error', 'loc': ('body', 3, 'ChatCompletionMessageCustomToolCallParam', 'type'), 'msg': "Input should be 'custom'", 'input': '', 'ctx': {'expected': "'custom'"}}

  File "/home/user1/vllm/vllm/entrypoints/serve/utils/api_utils.py", line 40, in create_chat_completion
    POST /v1/chat/completions [{'type': 'literal_error', 'loc': ('body', 3, 'ChatCompletionMessageFunctionToolCallParam', 'type'), 'msg': "Input should be 'function'", 'input': '', 'ctx': {'expected': "'function'"}}, {'type': 'missing', 'loc': ('body', 3, 'ChatCompletionMessageCustomToolCallParam', 'custom'), 'msg': 'Field required', 'input': {'function': {'arguments': '{"command": "dir', 'name': ''}, 'id': '', 'index': 3, 'type': ''}}, {'type': 'literal_error', 'loc': ('body', 3, 'ChatCompletionMessageCustomToolCallParam', 'type'), 'msg': "Input should be 'custom'", 'input': '', 'ctx': {'expected': "'custom'"}}]

This would instantly make me think of a template parsing/compiling issue from vLLMs side (again refer to ggml-org/llama.cpp#18675 ). The OAI API should work properly from Lates side otherwise other providers should have similar issues.

<|channel>thought
<channel|>

This also seems like a templating issue. Also check the corresponding --gemma-thinking flag if not yet done.

At this point i prompt "continue", but i'm not sure that the tool_call above actually got executed.

If the subagents task (and prompt processing) is not instant you should always be able to tab into it while it is running. If you cannot it definitely did not ran and would very much make me suspect the mentioned template issue (you can also check the raw JSON file, to see what the subagent returned, or if you can continue the session just asking the orchestrator what the subagent returned also works). While llama.cpp used to crash, vllm might catch the exception.

I will leave this open for now, so potentially new information can be shared and other users can see it. I (currently) suspect though that this is a vLLM issue.

[mpetruc]

@mlhher thank you for the feedback. My bet is also on the chat templating. I'll just close this for now.