[Bug]: llama4_pythonic tool parser fails with SyntaxError on nested list parameters

[mphilippnv]

Your current environment

I don't have direct access to the cluster the model is running in. But it's running on 8x H100 GPUs using TP 8, expert parallel.

This is the fp8 model from Huggingface.

These are the vllm serve args I'm using:

VLLM Version: 0.11.0

--port 8002 
--model /config/models/maverick 
--device cuda 
--tensor-parallel-size 8 
--disable-log-requests 
--max-num-batched-tokens 16000 
--served-model-name 'llama-4-maverick-17b-128e-instruct' 
--limit-mm-per-prompt image=50 
--kv-cache-dtype fp8 
--trust-remote-code 
--enable-auto-tool-choice 
--enable-chunked-prefill true 
--enable-prefix-caching 
--tool-call-parser llama4_pythonic 
--enable-expert-parallel 
--chat-template examples/tool_chat_template_llama4_pythonic.jinja 
--override-generation-config '{\"attn_temperature_tuning\": true}' 
--max-model-len 1000000

🐛 Describe the bug

Description

The llama4_pythonic tool parser intermittently fails to parse valid tool calls, resulting in:

1. SyntaxError from ast.parse() when model output is malformed (missing closing ])
2. Valid pythonic syntax returned as content instead of being parsed into tool_calls

Reproduction

Minimal curl (run 10+ times to observe intermittent failure):

curl -X POST https://your-vllm-endpoint/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-4-maverick-17b-128e-instruct",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "how do I enroll in benefits?"}
    ],
    "tools": [{
      "type": "function",
      "function": {
        "name": "enterprise_search",
        "description": "Search enterprise knowledge base",
        "parameters": {
          "type": "object",
          "properties": {
            "query": {"type": "string"},
            "rephrased_queries": {
              "type": "array",
              "items": {"type": "string"},
              "description": "List of 2 rephrased queries"
            }
          },
          "required": ["query", "rephrased_queries"]
        }
      }
    }],
    "tool_choice": "auto",
    "max_tokens": 500,
    "temperature": 0,
    "top_p": 0.95
  }'

Observed results (10 identical requests):

• 7/10: ✅ finish_reason: "tool_calls", properly parsed
• 3/10: ❌ finish_reason: "stop", pythonic syntax in content field, empty tool_calls

Failure Modes Observed

Mode 1: Valid pythonic not parsed

{
  "finish_reason": "stop",
  "message": {
    "content": "[enterprise_search(query=\"Benefits enrollment\", rephrased_queries=[\"...\", \"...\"])]",
    "tool_calls": []
  }
}

Parser fails to detect valid syntax → returned as content.

Mode 2: Model generates text after tool call

{
  "content": "[enterprise_search(...)]\n\nI was unable to execute this task..."
}

Model mixes tool call + text, which violates parser assumption.

Mode 3: Malformed output (missing bracket)

[enterprise_search(query='...', rephrased_queries=['...', '...'])

Model hits stop_reason: 200007 before completing → ast.parse() throws SyntaxError.

Suspected Root Cause

The below is suggested by Claude Opus 4.5 so take with a grain of salt.

1. Parser detection inconsistency - Valid pythonic output intermittently not recognized as tool call
2. No text-after-tool-call handling - Parser fails when model appends text after ]
3. Stop token interference - Model sometimes hits stop token (200007) mid-generation before completing brackets
4. Nested bracket complexity - Array parameters (rephrased_queries) create [...[...]...] nesting that may confuse detection

Error Logs

err.txt

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.