Skip to content
This repository was archived by the owner on Jul 4, 2025. It is now read-only.

Commit 30cef62

Browse files
gabrielle-onggabrielle-ong
committed
update models
1 parent 09434de commit 30cef62

File tree

1 file changed

+92
-48
lines changed

1 file changed

+92
-48
lines changed

‎README.md‎

Lines changed: 92 additions & 48 deletions
Original file line numberDiff line numberDiff line change
@@ -19,17 +19,26 @@
1919
- <a href="https://github.com/janhq/cortex.cpp/releases">Changelog</a> - <a href="https://github.com/janhq/cortex.cpp/issues">Bug reports</a> - <a href="https://discord.gg/AsJ8krTT3N">Discord</a>
2020
</p>
2121

22-
> ⚠️ **Cortex.cpp is currently in active development. This outlines the intended behavior of Cortex, which may not yet be fully implemented in the codebase.**
22+
> **Cortex.cpp is currently in active development.**
2323
2424
## Overview
2525

26-
Cortex.cpp is a Local AI engine that is used to run and customize LLMs. Cortex can be deployed as a standalone server, or integrated into apps like [Jan.ai](https://jan.ai/).
26+
Cortex is a Local AI API Platform that is used to run and customize LLMs.
2727

28-
Cortex.cpp is a multi-engine that uses `llama.cpp` as the default engine but also supports the following:
28+
Key Features:
29+
- Straightforward CLI (inspired by Ollama)
30+
- Full C++ implementation, packageable into Desktop and Mobile apps
31+
- Pull from Huggingface of Cortex Built-in Model Library
32+
- Models stored in universal file formats (vs blobs)
33+
- Swappable Engines (default: [`llamacpp`](https://github.com/janhq/cortex.llamacpp), future: [`ONNXRuntime`](https://github.com/janhq/cortex.onnx), [`TensorRT-LLM`](https://github.com/janhq/cortex.tensorrt-llm))
34+
- Cortex can be deployed as a standalone API server, or integrated into apps like [Jan.ai](https://jan.ai/)
2935

30-
- [`llamacpp`](https://github.com/janhq/cortex.llamacpp)
31-
- [`onnx`](https://github.com/janhq/cortex.onnx)
32-
- [`tensorrt-llm`](https://github.com/janhq/cortex.tensorrt-llm)
36+
Cortex's roadmap is to implement the full OpenAI API including Tools, Runs, Multi-modal and Realtime APIs.
37+
38+
Quick links:
39+
- Documentation: https://cortex.so/
40+
- API Reference: https://cortex.so/api-reference/
41+
- Discord Community: https://discord.gg/Exe46xPMbK
3342

3443
## Local Installation
3544

@@ -59,47 +68,82 @@ Cortex also has a [Network Installer](#network-installer) which downloads the ne
5968

6069
```bash
6170
sudo apt install ./cortex-local-installer.deb
62-
# or
63-
sudo apt install ./cortex-network-installer.deb
6471
```
6572

6673
- The binary will be installed in the `/usr/bin/` directory.
6774

6875
## Usage
6976

70-
After installation, you can run Cortex.cpp from the command line by typing `cortex --help`. For Beta preview, you can run `cortex-beta --help`.
77+
### CLI
78+
79+
After installation, you can run Cortex.cpp from the command line by typing `cortex --help`.
80+
81+
```
82+
cortex pull llama3.2
83+
cortex pull bartowski/Meta-Llama-3.1-8B-Instruct-GGUF
84+
cortex run llama3.2
85+
cortex models ps
86+
cortex models stop llama3.2
87+
cortex stop
88+
```
89+
90+
Refer to our [Quickstart](https://cortex.so/docs/quickstart/) and
91+
[CLI documentation](https://cortex.so/docs/cli) for more details.
7192

72-
## Built-in Model Library
93+
### API:
94+
Cortex.cpp includes a REST API accessible at `localhost:39281`.
95+
96+
Refer to our [API documentation](https://cortex.so/api-reference) for more details
97+
98+
## Models & Quantizations
99+
100+
Cortex.cpp allows users to pull models from multiple Model Hubs, offering flexibility and extensive model access.
101+
102+
Currently Cortex supports pulling from:
103+
- Hugging Face: GGUF models eg `author/Model-GGUF`
104+
- Cortex Built-in Models
73105

74-
Cortex.cpp supports various models available on the [Cortex Hub](https://huggingface.co/cortexso).
75106
Once downloaded, the model `.gguf` and `model.yml` files are stored in `~\cortexcpp\models`.
76107

77-
Example models:
108+
> **Note**:
109+
> You should have at least 8 GB of RAM available to run the 7B models, 16 GB to run the 14B models, and 32 GB to run the 32B models.
110+
111+
### Cortex Model Hub & Quantizations
78112

79113
| Model /Engine | llama.cpp | Command |
80114
| -------------- | --------------------- | ----------------------------- |
115+
| phi-3.5 | ✅ | cortex run phi3.5 |
116+
| llama3.2 | ✅ | cortex run llama3.1 |
81117
| llama3.1 | ✅ | cortex run llama3.1 |
82-
| llama3 | ✅ | cortex run llama3 |
83-
| mistral | ✅ | cortex run mistral |
84-
| qwen2 | ✅ | cortex run qwen2 |
85118
| codestral | ✅ | cortex run codestral |
86-
| command-r | ✅ | cortex run command-r |
87-
| gemma | ✅ | cortex run gemma |
88-
| mixtral | ✅ | cortex run mixtral |
119+
| gemma2 | ✅ | cortex run gemma2 |
120+
| mistral | ✅ | cortex run mistral |
121+
| ministral | ✅ | cortex run ministral |
122+
| qwen2 | ✅ | cortex run qwen2.5 |
89123
| openhermes-2.5 | ✅ | cortex run openhermes-2.5 |
90-
| phi3 | ✅ | cortex run phi3 |
91124
| tinyllama | ✅ | cortex run tinyllama |
92125

93-
> **Note**:
94-
> You should have at least 8 GB of RAM available to run the 7B models, 16 GB to run the 14B models, and 32 GB to run the 32B models.
95-
96-
## Cortex.cpp CLI Commands
126+
View all [Cortex Built-in Models](https://cortex.so/models).
97127

98-
For complete details on CLI commands, please refer to our [CLI documentation](https://cortex.so/docs/cli).
99-
100-
## REST API
101-
102-
Cortex.cpp includes a REST API accessible at `localhost:39281`. For a complete list of endpoints and their usage, visit our [API documentation](https://cortex.so/api-reference).
128+
Cortex supports multiple quantizations for each model.
129+
```
130+
❯ cortex-nightly pull llama3.2
131+
Downloaded models:
132+
llama3.2:3b-gguf-q2-k
133+
134+
Available to download:
135+
1. llama3.2:3b-gguf-q3-kl
136+
2. llama3.2:3b-gguf-q3-km
137+
3. llama3.2:3b-gguf-q3-ks
138+
4. llama3.2:3b-gguf-q4-km (default)
139+
5. llama3.2:3b-gguf-q4-ks
140+
6. llama3.2:3b-gguf-q5-km
141+
7. llama3.2:3b-gguf-q5-ks
142+
8. llama3.2:3b-gguf-q6-k
143+
9. llama3.2:3b-gguf-q8-0
144+
145+
Select a model (1-9):
146+
```
103147

104148
## Advanced Installation
105149

@@ -129,13 +173,17 @@ Cortex.cpp is available with a Network Installer, which is a smaller installer b
129173
### Beta & Nightly Versions
130174

131175
Cortex releases 2 preview versions for advanced users to try new features early (we appreciate your feedback!)
132-
- Beta (early preview)
133-
- Nightly (released every night). Nightly pulls the latest changes from upstream [llama.cpp](https://github.com/ggerganov/llama.cpp/) repo, creates a PR and runs our tests. If all test pass, the PR is automatically merged into our repo, with the latest llama.cpp version.
176+
- Beta (early preview)
177+
- CLI command: `cortex-beta`
178+
- Nightly (released every night)
179+
- CLI Command: `cortex-nightly`
180+
- Nightly automatically pulls the latest changes from upstream [llama.cpp](https://github.com/ggerganov/llama.cpp/) repo, creates a PR and runs tests.
181+
- If all test pass, the PR is automatically merged into our repo, with the latest llama.cpp version.
134182

135183
#### Local Installer (Default)
136184
<table>
137185
<tr style="text-align:center">
138-
<td style="text-align:center"><b>Version Type</b></td>
186+
<td style="text-align:center"><b>Version</b></td>
139187
<td style="text-align:center"><b>Windows</b></td>
140188
<td style="text-align:center"><b>MacOS</b></td>
141189
<td style="text-align:center"><b>Linux</b></td>
@@ -145,19 +193,19 @@ Cortex releases 2 preview versions for advanced users to try new features early
145193
<td style="text-align:center">
146194
<a href='https://app.cortexcpp.com/download/beta/windows-amd64-local'>
147195
<img src='https://github.com/janhq/docs/blob/main/static/img/windows.png' style="height:14px; width: 14px" />
148-
<b>cortex-beta-windows-local-installer.exe</b>
196+
cortex-beta-windows-local-installer.exe
149197
</a>
150198
</td>
151199
<td style="text-align:center">
152200
<a href='https://app.cortexcpp.com/download/beta/mac-universal-local'>
153201
<img src='https://github.com/janhq/docs/blob/main/static/img/mac.png' style="height:15px; width: 15px" />
154-
<b>cortex-beta-mac-local-installer.pkg</b>
202+
cortex-beta-mac-local-installer.pkg
155203
</a>
156204
</td>
157205
<td style="text-align:center">
158206
<a href='https://app.cortexcpp.com/download/beta/linux-amd64-local'>
159207
<img src='https://github.com/janhq/docs/blob/main/static/img/linux.png' style="height:14px; width: 14px" />
160-
<b>cortex-beta-linux-local-installer.deb</b>
208+
cortex-beta-linux-local-installer.deb
161209
</a>
162210
</td>
163211
</tr>
@@ -166,19 +214,19 @@ Cortex releases 2 preview versions for advanced users to try new features early
166214
<td style="text-align:center">
167215
<a href='https://app.cortexcpp.com/download/nightly/windows-amd64-local'>
168216
<img src='https://github.com/janhq/docs/blob/main/static/img/windows.png' style="height:14px; width: 14px" />
169-
<b>cortex-nightly-windows-local-installer.exe</b>
217+
cortex-nightly-windows-local-installer.exe
170218
</a>
171219
</td>
172220
<td style="text-align:center">
173221
<a href='https://app.cortexcpp.com/download/nightly/mac-universal-local'>
174222
<img src='https://github.com/janhq/docs/blob/main/static/img/mac.png' style="height:15px; width: 15px" />
175-
<b>cortex-nightly-mac-local-installer.pkg</b>
223+
cortex-nightly-mac-local-installer.pkg
176224
</a>
177225
</td>
178226
<td style="text-align:center">
179227
<a href='https://app.cortexcpp.com/download/nightly/linux-amd64-local'>
180228
<img src='https://github.com/janhq/docs/blob/main/static/img/linux.png' style="height:14px; width: 14px" />
181-
<b>cortex-nightly-linux-local-installer.deb</b>
229+
cortex-nightly-linux-local-installer.deb
182230
</a>
183231
</td>
184232
</tr>
@@ -198,19 +246,19 @@ Cortex releases 2 preview versions for advanced users to try new features early
198246
<td style="text-align:center">
199247
<a href='https://app.cortexcpp.com/download/beta/windows-amd64-network'>
200248
<img src='https://github.com/janhq/docs/blob/main/static/img/windows.png' style="height:14px; width: 14px" />
201-
<b>cortex-beta-windows-network-installer.exe</b>
249+
cortex-beta-windows-network-installer.exe
202250
</a>
203251
</td>
204252
<td style="text-align:center">
205253
<a href='https://app.cortexcpp.com/download/beta/mac-universal-network'>
206254
<img src='https://github.com/janhq/docs/blob/main/static/img/mac.png' style="height:15px; width: 15px" />
207-
<b>cortex-beta-mac-network-installer.pkg</b>
255+
cortex-beta-mac-network-installer.pkg
208256
</a>
209257
</td>
210258
<td style="text-align:center">
211259
<a href='https://app.cortexcpp.com/download/beta/linux-amd64-network'>
212260
<img src='https://github.com/janhq/docs/blob/main/static/img/linux.png' style="height:15px; width: 15px" />
213-
<b>cortex-beta-linux-network-installer.deb</b>
261+
cortex-beta-linux-network-installer.deb
214262
</a>
215263
</td>
216264
</tr>
@@ -219,19 +267,19 @@ Cortex releases 2 preview versions for advanced users to try new features early
219267
<td style="text-align:center">
220268
<a href='https://app.cortexcpp.com/download/nightly/windows-amd64-network'>
221269
<img src='https://github.com/janhq/docs/blob/main/static/img/windows.png' style="height:15px; width: 15px" />
222-
<b>cortex-nightly-windows-network-installer.exe</b>
270+
cortex-nightly-windows-network-installer.exe
223271
</a>
224272
</td>
225273
<td style="text-align:center">
226274
<a href='https://app.cortexcpp.com/download/nightly/mac-universal-network'>
227275
<img src='https://github.com/janhq/docs/blob/main/static/img/mac.png' style="height:15px; width: 15px" />
228-
<b>cortex-nightly-mac-network-installer.pkg</b>
276+
cortex-nightly-mac-network-installer.pkg
229277
</a>
230278
</td>
231279
<td style="text-align:center">
232280
<a href='https://app.cortexcpp.com/download/nightly/linux-amd64-network'>
233281
<img src='https://github.com/janhq/docs/blob/main/static/img/linux.png' style="height:15px; width: 15px" />
234-
<b>cortex-nightly-linux-network-installer.deb</b>
282+
cortex-nightly-linux-network-installer.deb
235283
</a>
236284
</td>
237285
</tr>
@@ -263,7 +311,6 @@ cmake .. -DBUILD_SHARED_LIBS=OFF -DCMAKE_TOOLCHAIN_FILE=path_to_vcpkg_folder/vcp
263311
6. Verify that Cortex.cpp is installed correctly by getting help information.
264312

265313
```sh
266-
# Get the help information
267314
cortex -h
268315
```
269316

@@ -292,7 +339,6 @@ make -j4
292339
6. Verify that Cortex.cpp is installed correctly by getting help information.
293340

294341
```sh
295-
# Get the help information
296342
cortex -h
297343
```
298344

@@ -321,8 +367,7 @@ make -j4
321367
6. Verify that Cortex.cpp is installed correctly by getting help information.
322368

323369
```sh
324-
# Get help
325-
cortex
370+
cortex -h
326371
```
327372

328373
## Uninstallation
@@ -346,7 +391,6 @@ For MacOS, there is a uninstaller script comes with the binary and added to the
346391
### Linux
347392

348393
```bash
349-
# For stable builds
350394
sudo apt remove cortexcpp
351395
```
352396

0 commit comments

Comments
 (0)