"Failed to load chat template..." when using embedding gguf model (not an instruct chat model) that has no tokenizer.chat_template key (embedding model do not require chat template)
Paddler's agent startup unconditionally requires a chat template (either from the model's GGUF metadata or from an override) and when no override is configured, model.chat_template(None) fails, aborting agent initialization.
The root cause is traced to paddler_agent/src/continuous_batch_arbiter.rs:136-149:
let llama_chat_template_string = match chat_template_override {
Some(chat_template) => chat_template.content, // <-- ONLY path that works
None => model
.chat_template(None) // <-- FAILS: no tokenizer.chat_template in GGUF
.context("Failed to load chat template...")?
.to_string()?,
};
if chat_template_loaded_tx.send(()).is_err() { // <-- never reached if above fails
// ...
}
The error chain is:
1. chat_template_override is None (because use_chat_template_override: false or not set)
2. model.chat_template(None) fails ; the GGUF has no tokenizer.chat_template metadata
3. The ? operator returns the error, thread exits before chat_template_loaded_tx.send(())
4. Main async receives: "Failed to receive chat template loaded signal"
5. Then: "Scheduler thread did not signal agent-warm-and-scheduler-running before exiting"
It is NOT an architecture incompatibility per se ; Paddler's embedding processor (continuous_batch_embedding_processor.rs) uses llama_context.
embeddings_seq_ith() which works with bidirectional models. The blocking issue is the mandatory chat template loading during agent startup. Paddler has tests ( https://github.com/intentee/paddler/tree/main/paddler_tests/tests) confirming this pattern:
- balancer_reports_unable_to_find_chat_template_for_embedding_model.rs
- chat_template_override_applied_to_embedding_model.rs
"Failed to load chat template..." when using embedding gguf model (not an instruct chat model) that has no tokenizer.chat_template key (embedding model do not require chat template)
Paddler's agent startup unconditionally requires a chat template (either from the model's GGUF metadata or from an override) and when no override is configured, model.chat_template(None) fails, aborting agent initialization.
The root cause is traced to paddler_agent/src/continuous_batch_arbiter.rs:136-149: