Skip to content

Chat template from chat_template.jinja for all possible paths #4055

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
dkalinowski wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Chat template from chat_template.jinja for all possible paths #4055

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
831a375
save
dkalinowski Mar 12, 2026
15dae46
save
dkalinowski Mar 12, 2026
1a7437e
save
dkalinowski Mar 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions src/llm/language_model/continuous_batching/servable_initializer.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -210,6 +210,21 @@ Status ContinuousBatchingServableInitializer::initialize(std::shared_ptr<GenAiSe
properties->schedulerConfig, properties->device,
properties->pluginConfig, properties->tokenizerPluginConfig);
properties->tokenizer = properties->pipeline->get_tokenizer();

// Override chat template from chat_template.jinja file if present in model directory
std::filesystem::path chatTemplateJinjaPath = std::filesystem::path(parsedModelsPath) / "chat_template.jinja";
Copy link

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

chatTemplateJinjaPath is derived as parsedModelsPath / "chat_template.jinja". Since parseModelsPath() accepts .gguf files as a valid models_path, this won’t locate a template stored next to a GGUF file (it will incorrectly search under <file>.gguf/chat_template.jinja). Consider basing the search directory on parsedModelsPath if it’s a directory, otherwise use parent_path().

Suggested change
std::filesystem::path chatTemplateJinjaPath = std::filesystem::path(parsedModelsPath) / "chat_template.jinja";
std::filesystem::path modelsPathFs(parsedModelsPath);
std::filesystem::path chatTemplateDir = modelsPathFs;
if (!chatTemplateDir.empty() && !std::filesystem::is_directory(chatTemplateDir)) {
chatTemplateDir = chatTemplateDir.parent_path();
}
std::filesystem::path chatTemplateJinjaPath = chatTemplateDir / "chat_template.jinja";

Copilot uses AI. Check for mistakes.
if (std::filesystem::exists(chatTemplateJinjaPath)) {
std::ifstream chatTemplateFile(chatTemplateJinjaPath);
if (chatTemplateFile.is_open()) {
std::string chatTemplateContent((std::istreambuf_iterator<char>(chatTemplateFile)),
std::istreambuf_iterator<char>());
if (!chatTemplateContent.empty()) {
Comment on lines +218 to +221
Copy link

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This code uses std::istreambuf_iterator, but there is no explicit #include <iterator> in this translation unit. Please add it to ensure the file compiles without relying on indirect includes.

Copilot uses AI. Check for mistakes.
properties->tokenizer.set_chat_template(chatTemplateContent);
}
} else {
SPDLOG_LOGGER_WARN(llm_calculator_logger, "Failed to open chat template file: {}", chatTemplateJinjaPath.string());
}
}
} catch (const std::exception& e) {
SPDLOG_ERROR("Error during llm node initialization for models_path: {} exception: {}", parsedModelsPath, e.what());
return StatusCode::LLM_NODE_RESOURCE_STATE_INITIALIZATION_FAILED;
Expand Down
17 changes: 17 additions & 0 deletions src/llm/language_model/legacy/servable_initializer.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
//*****************************************************************************
#include <fstream>
#include <memory>
#include <stdexcept>
#include <string>
Expand Down Expand Up @@ -100,6 +101,22 @@ Status LegacyServableInitializer::initialize(std::shared_ptr<GenAiServable>& ser
try {
properties->pipeline = std::make_shared<ov::genai::LLMPipeline>(parsedModelsPath, properties->device, properties->pluginConfig);
properties->tokenizer = properties->pipeline->get_tokenizer();

// Override chat template from chat_template.jinja file if present in model directory
std::filesystem::path chatTemplateJinjaPath = std::filesystem::path(parsedModelsPath) / "chat_template.jinja";
Copy link

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

chatTemplateJinjaPath is built by appending chat_template.jinja to parsedModelsPath. However parseModelsPath() allows models_path to be either a directory or a .gguf file; when it’s a file, this will look for <file>.gguf/chat_template.jinja and never find the template next to the model. Consider resolving the “template directory” as parsedModelsPath when it’s a directory, otherwise parent_path() (and use that for the lookup).

Suggested change
std::filesystem::path chatTemplateJinjaPath = std::filesystem::path(parsedModelsPath) / "chat_template.jinja";
std::filesystem::path modelsPathFs(parsedModelsPath);
std::filesystem::path chatTemplateDir =
std::filesystem::is_directory(modelsPathFs) ? modelsPathFs : modelsPathFs.parent_path();
std::filesystem::path chatTemplateJinjaPath = chatTemplateDir / "chat_template.jinja";

Copilot uses AI. Check for mistakes.
if (std::filesystem::exists(chatTemplateJinjaPath)) {
std::ifstream chatTemplateFile(chatTemplateJinjaPath);
if (chatTemplateFile.is_open()) {
std::string chatTemplateContent((std::istreambuf_iterator<char>(chatTemplateFile)),
std::istreambuf_iterator<char>());
if (!chatTemplateContent.empty()) {
Comment on lines +109 to +112
Copy link

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This block uses std::istreambuf_iterator (and std::filesystem::path), but the file doesn’t include the corresponding standard headers (<iterator> / <filesystem>). Please add the missing includes to avoid relying on transitive headers (can break on different libstdc++/libc++ versions).

Copilot uses AI. Check for mistakes.
properties->tokenizer.set_chat_template(chatTemplateContent);
SPDLOG_LOGGER_DEBUG(llm_calculator_logger, "Loaded custom chat template from: {}", chatTemplateJinjaPath.string());
}
} else {
SPDLOG_LOGGER_WARN(llm_calculator_logger, "Failed to open chat template file: {}", chatTemplateJinjaPath.string());
}
}
Comment on lines +104 to +119
Copy link

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The same “read chat_template.jinja and set tokenizer chat template” logic is duplicated across multiple servable initializers in this PR. To prevent drift (e.g., different logging/edge-case handling), consider extracting it into a shared helper (e.g., on GenAiServableInitializer).

Suggested change
// Override chat template from chat_template.jinja file if present in model directory
std::filesystem::path chatTemplateJinjaPath = std::filesystem::path(parsedModelsPath) / "chat_template.jinja";
if (std::filesystem::exists(chatTemplateJinjaPath)) {
std::ifstream chatTemplateFile(chatTemplateJinjaPath);
if (chatTemplateFile.is_open()) {
std::string chatTemplateContent((std::istreambuf_iterator<char>(chatTemplateFile)),
std::istreambuf_iterator<char>());
if (!chatTemplateContent.empty()) {
properties->tokenizer.set_chat_template(chatTemplateContent);
SPDLOG_LOGGER_DEBUG(llm_calculator_logger, "Loaded custom chat template from: {}", chatTemplateJinjaPath.string());
}
} else {
SPDLOG_LOGGER_WARN(llm_calculator_logger, "Failed to open chat template file: {}", chatTemplateJinjaPath.string());
}
}

Copilot uses AI. Check for mistakes.
} catch (const std::exception& e) {
SPDLOG_ERROR("Error during llm node initialization for models_path: {} exception: {}", parsedModelsPath, e.what());
return StatusCode::LLM_NODE_RESOURCE_STATE_INITIALIZATION_FAILED;
Expand Down
17 changes: 17 additions & 0 deletions src/llm/visual_language_model/legacy/servable_initializer.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
//*****************************************************************************
#include <fstream>
#include <memory>
#include <stdexcept>
#include <string>
Expand Down Expand Up @@ -84,6 +85,22 @@ Status VisualLanguageModelLegacyServableInitializer::initialize(std::shared_ptr<
try {
properties->pipeline = std::make_shared<ov::genai::VLMPipeline>(parsedModelsPath, properties->device, properties->pluginConfig);
properties->tokenizer = properties->pipeline->get_tokenizer();

// Override chat template from chat_template.jinja file if present in model directory
std::filesystem::path chatTemplateJinjaPath = std::filesystem::path(parsedModelsPath) / "chat_template.jinja";
if (std::filesystem::exists(chatTemplateJinjaPath)) {
std::ifstream chatTemplateFile(chatTemplateJinjaPath);
if (chatTemplateFile.is_open()) {
std::string chatTemplateContent((std::istreambuf_iterator<char>(chatTemplateFile)),
std::istreambuf_iterator<char>());
if (!chatTemplateContent.empty()) {
Comment on lines +90 to +96
Copy link

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This block uses std::istreambuf_iterator / std::filesystem::path, but the file doesn’t include the corresponding standard headers (<iterator> / <filesystem>). Please add the missing includes to avoid relying on transitive headers.

Copilot uses AI. Check for mistakes.
properties->tokenizer.set_chat_template(chatTemplateContent);
SPDLOG_LOGGER_DEBUG(llm_calculator_logger, "Loaded custom chat template from: {}", chatTemplateJinjaPath.string());
}
} else {
SPDLOG_LOGGER_WARN(llm_calculator_logger, "Failed to open chat template file: {}", chatTemplateJinjaPath.string());
}
}
} catch (const std::exception& e) {
SPDLOG_ERROR("Error during llm node initialization for models_path: {} exception: {}", parsedModelsPath, e.what());
return StatusCode::LLM_NODE_RESOURCE_STATE_INITIALIZATION_FAILED;
Expand Down
Loading