|
2 | 2 | slug: /engines |
3 | 3 | title: Engines |
4 | 4 | --- |
5 | | -import DocCardList from '@theme/DocCardList'; |
6 | 5 |
|
| 6 | +import DocCardList from "@theme/DocCardList"; |
7 | 7 |
|
8 | 8 | :::warning |
9 | 9 | 🚧 Cortex.cpp is currently under development. Our documentation outlines the intended behavior of Cortex, which may not yet be fully implemented in the codebase. |
10 | 10 | ::: |
11 | 11 |
|
| 12 | +# Engines |
| 13 | + |
| 14 | +Engines in Cortex serve as execution drivers for machine learning models, providing the runtime environment necessary for model operations. Each engine is specifically designed to optimize the performance and ensure compatibility with its corresponding model types. |
| 15 | + |
| 16 | +## Supported Engines |
| 17 | + |
| 18 | +Cortex currently supports three industry-standard engines: |
| 19 | + |
| 20 | +| Engine | Source | Description | |
| 21 | +| -------------------------------------------------------- | --------- | -------------------------------------------------------------------------------------- | |
| 22 | +| [llama.cpp](https://github.com/ggerganov/llama.cpp) | ggerganov | Inference of Meta's LLaMA model (and others) in pure C/C++ | |
| 23 | +| [ONNX Runtime](https://github.com/microsoft/onnxruntime) | Microsoft | ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator | |
| 24 | +| [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM) | NVIDIA | GPU-optimized inference engine for large language models | |
| 25 | + |
| 26 | +> **Note:** Cortex also supports users in building their own engines. |
| 27 | +
|
| 28 | +## Features |
| 29 | + |
| 30 | +- **Engine Retrieval**: Install engines with a single click. |
| 31 | +- **Engine Management**: Easily manage engines by type, variant, and version. |
| 32 | +- **User-Friendly Interface**: Access models via Command Line Interface (CLI) or HTTP API. |
| 33 | +- **Engine Selection**: Select the appropriate engines to run your models. |
| 34 | + |
| 35 | +## Usage |
| 36 | + |
| 37 | +Cortex offers comprehensive support for multiple engine types, including [llama.cpp](https://github.com/ggerganov/llama.cpp), [ONNX Runtime](https://github.com/microsoft/onnxruntime), and [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM). These engines are utilized to load their corresponding model types. The platform provides a flexible management system for different engine variants and versions, enabling developers and users to easily rollback changes or compare performance metrics across different engine versions. |
| 38 | + |
| 39 | +### Installing an engine |
| 40 | + |
| 41 | +Cortex makes it extremely easy to install an engine. For example, to run a `GGUF` model, you will need the `llama-cpp` engine. To install it, simply enter `cortex engines install llama-cpp` into your terminal and wait for the process to complete. Cortex will automatically pull the latest stable version suitable for your PC's specifications. |
| 42 | + |
| 43 | +#### CLI |
| 44 | + |
| 45 | +To install an engine using the CLI, use the following command: |
| 46 | + |
| 47 | +```sh |
| 48 | +cortex engines install llama-cpp |
| 49 | +Validating download items, please wait.. |
| 50 | +Start downloading.. |
| 51 | +llama-cpp 100%[==================================================] [00m:00s] 1.24 MB/1.24 MB |
| 52 | +Engine llama-cpp downloaded successfully! |
| 53 | +``` |
| 54 | + |
| 55 | +#### HTTP API |
| 56 | + |
| 57 | +To install an engine using the HTTP API, use the following command: |
| 58 | + |
| 59 | +```sh |
| 60 | +curl --location --request POST 'http://127.0.0.1:39281/engines/install/llama-cpp' |
| 61 | +``` |
| 62 | + |
| 63 | +Example response: |
| 64 | + |
| 65 | +```json |
| 66 | +{ |
| 67 | + "message": "Engine llama-cpp starts installing!" |
| 68 | +} |
| 69 | +``` |
| 70 | + |
| 71 | +### Listing engines |
| 72 | + |
| 73 | +Cortex allowing clients to easily list current engines and their statuses. Each engine type can have different variants and versions, which are crucial for debugging and performance optimization. Different variants cater to specific hardware configurations, such as CUDA for NVIDIA GPUs and Vulkan for AMD GPUs on Windows, or AVX512 support for CPUs. |
| 74 | + |
| 75 | +#### CLI |
| 76 | + |
| 77 | +You can list the available engines using the following command: |
| 78 | + |
| 79 | +```sh |
| 80 | +cortex engines list |
| 81 | ++---+--------------+-------------------+---------+-----------+--------------+ |
| 82 | +| # | Name | Supported Formats | Version | Variant | Status | |
| 83 | ++---+--------------+-------------------+---------+-----------+--------------+ |
| 84 | +| 1 | onnxruntime | ONNX | | | Incompatible | |
| 85 | ++---+--------------+-------------------+---------+-----------+--------------+ |
| 86 | +| 2 | llama-cpp | GGUF | 0.1.37 | mac-arm64 | Ready | |
| 87 | ++---+--------------+-------------------+---------+-----------+--------------+ |
| 88 | +| 3 | tensorrt-llm | TensorRT Engines | | | Incompatible | |
| 89 | ++---+--------------+-------------------+---------+-----------+--------------+ |
| 90 | +``` |
| 91 | + |
| 92 | +#### HTTP API |
| 93 | + |
| 94 | +You can also retrieve the list of engines via the HTTP API: |
| 95 | + |
| 96 | +```sh |
| 97 | +curl --location 'http://127.0.0.1:39281/v1/engines' |
| 98 | +``` |
| 99 | + |
| 100 | +Example response: |
| 101 | + |
| 102 | +```json |
| 103 | +{ |
| 104 | + "data": [ |
| 105 | + { |
| 106 | + "description": "This extension enables chat completion API calls using the Onnx engine", |
| 107 | + "format": "ONNX", |
| 108 | + "name": "onnxruntime", |
| 109 | + "productName": "onnxruntime", |
| 110 | + "status": "Incompatible", |
| 111 | + "variant": "", |
| 112 | + "version": "" |
| 113 | + }, |
| 114 | + { |
| 115 | + "description": "This extension enables chat completion API calls using the LlamaCPP engine", |
| 116 | + "format": "GGUF", |
| 117 | + "name": "llama-cpp", |
| 118 | + "productName": "llama-cpp", |
| 119 | + "status": "Ready", |
| 120 | + "variant": "mac-arm64", |
| 121 | + "version": "0.1.37" |
| 122 | + }, |
| 123 | + { |
| 124 | + "description": "This extension enables chat completion API calls using the TensorrtLLM engine", |
| 125 | + "format": "TensorRT Engines", |
| 126 | + "name": "tensorrt-llm", |
| 127 | + "productName": "tensorrt-llm", |
| 128 | + "status": "Incompatible", |
| 129 | + "variant": "", |
| 130 | + "version": "" |
| 131 | + } |
| 132 | + ], |
| 133 | + "object": "list", |
| 134 | + "result": "OK" |
| 135 | +} |
| 136 | +``` |
| 137 | + |
| 138 | +### Getting detail information of an engine |
| 139 | + |
| 140 | +Cortex allows users to retrieve detailed information about a specific engine. This includes supported formats, versions, variants, and status. This feature helps users understand the capabilities and compatibility of the engine they are working with. |
| 141 | + |
| 142 | +#### CLI |
| 143 | + |
| 144 | +To retrieve detailed information about an engine using the CLI, use the following command: |
| 145 | + |
| 146 | +```sh |
| 147 | +cortex engines get llama-cpp |
| 148 | ++-----------+-------------------+---------+-----------+--------+ |
| 149 | +| Name | Supported Formats | Version | Variant | Status | |
| 150 | ++-----------+-------------------+---------+-----------+--------+ |
| 151 | +| llama-cpp | GGUF | 0.1.37 | mac-arm64 | Ready | |
| 152 | ++-----------+-------------------+---------+-----------+--------+ |
| 153 | +``` |
| 154 | + |
| 155 | +This command will display information such as the engine's name, supported formats, version, variant, and status. |
| 156 | + |
| 157 | +#### HTTP API |
| 158 | + |
| 159 | +To retrieve detailed information about an engine using the HTTP API, send a GET request to the appropriate endpoint: |
| 160 | + |
| 161 | +```sh |
| 162 | +curl --location 'http://127.0.0.1:39281/engines/llama-cpp' |
| 163 | +``` |
| 164 | + |
| 165 | +This request will return a JSON response containing detailed information about the engine, including its description, format, name, product name, status, variant, and version. |
| 166 | +Example response: |
| 167 | + |
| 168 | +```json |
| 169 | +{ |
| 170 | + "description": "This extension enables chat completion API calls using the LlamaCPP engine", |
| 171 | + "format": "GGUF", |
| 172 | + "name": "llama-cpp", |
| 173 | + "productName": "llama-cpp", |
| 174 | + "status": "Not Installed", |
| 175 | + "variant": "", |
| 176 | + "version": "" |
| 177 | +} |
| 178 | +``` |
| 179 | + |
| 180 | +### Uninstalling an engine |
| 181 | + |
| 182 | +Cortex provides an easy way to uninstall an engine. This is useful when users want to uninstall the current version and then install the latest stable version of a particular engine. |
| 183 | + |
| 184 | +#### CLI |
| 185 | + |
| 186 | +To uninstall an engine, use the following CLI command: |
| 187 | + |
| 188 | +```sh |
| 189 | +cortex engines uninstall llama-cpp |
| 190 | +``` |
| 191 | + |
| 192 | +#### HTTP API |
| 193 | + |
| 194 | +To uninstall an engine using the HTTP API, send a DELETE request to the appropriate endpoint. |
| 195 | + |
| 196 | +```sh |
| 197 | +curl --location --request DELETE 'http://127.0.0.1:39281/engines/llama-cpp' |
| 198 | +``` |
| 199 | + |
| 200 | +Example response: |
| 201 | + |
| 202 | +```json |
| 203 | +{ |
| 204 | + "message": "Engine llama-cpp uninstalled successfully!" |
| 205 | +} |
| 206 | +``` |
| 207 | + |
| 208 | +### Upcoming Engine Features |
| 209 | + |
| 210 | +- Enhanced engine update mechanism with automated compatibility checks |
| 211 | +- Seamless engine switching between variants and versions |
| 212 | +- Improved Vulkan engine support with optimized performance |
| 213 | + |
12 | 214 | <DocCardList /> |
0 commit comments