Skip to content
This repository was archived by the owner on Jul 4, 2025. It is now read-only.

Commit 1d53a8e

Browse files
chore: add document for model vs engine
1 parent fd2a475 commit 1d53a8e

File tree

2 files changed

+6
-3
lines changed

2 files changed

+6
-3
lines changed

docs/docs/capabilities/models/index.mdx

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,10 @@ description: The Model section overview
77
🚧 Cortex.cpp is currently under development. Our documentation outlines the intended behavior of Cortex, which may not yet be fully implemented in the codebase.
88
:::
99

10+
Models in cortex.cpp are used for inference purposes (e.g., chat completion, embedding, etc.). We support two types of models: local and remote.
11+
Local models use a local inference engine to run completely offline on your hardware. Currently, we support llama.cpp with the GGUF model format, and we have plans to support TensorRT-LLM and ONNX engines in the future.
12+
Remote models (like OpenAI GPT-4 and Claude 3.5 Sonnet) use remote engines. Support for OpenAI and Anthropic engines is under development and will be available in cortex.cpp soon.
13+
1014
When Cortex.cpp is started, it automatically starts an API server, this is inspired by Docker CLI. This server manages various model endpoints. These endpoints facilitate the following:
1115
- **Model Operations**: Run and stop models.
1216
- **Model Management**: Manage your local models.

docs/docs/guides/function-calling.md

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,9 @@
11
---
22
title: Function Calling
33
---
4-
54
# Function calling with OpenAI compatible
65

7-
This tutorial, I use the `mistral-nemo:12b-gguf-q4-km` for testing with cortex.cpp. All steps are reproduced from original openai instruction https://platform.openai.com/docs/guides/function-calling
6+
This tutorial, I use the `mistral-nemo:12b-gguf-q4-km` for testing function calling with cortex.cpp. All steps are reproduced from original openai instruction https://platform.openai.com/docs/guides/function-calling
87

98
## Step by step with function calling
109

@@ -113,7 +112,7 @@ completion_payload = {
113112
"messages": [
114113
{"role": "system", "content": "You are a helpful customer support assistant. Use the supplied tools to assist the user."},
115114
{"role": "user", "content": "Hi, can you tell me the delivery date for my order?"},
116-
{"role": "assistant", "content": "Sure! Could you please provide your order ID so I can look up the delivery date for you?"},
115+
{"role": "assistant", "content": "Of course! Please provide your order ID so I can look it up."},
117116
{"role": "user", "content": "i think it is order_12345"},
118117
]
119118
}

0 commit comments

Comments
 (0)