v1.0.2 QA checklist (6 Nov)

Author: gabrielle-ong

.github/ISSUE_TEMPLATE/QA_checklist.md

@@ -17,43 +17,49 @@ OS (select one)
 
 --------
 
-# 1. Manual QA
+# 1. Manual QA (CLI)
 ## Installation
+- [ ]  it should install with local installer (default; no internet required during installation, all dependencies bundled)
 - [ ]  it should install with network installer
-- [ ]  it should install with local installer
-- [ ] it should install 2 binaries (cortex and cortex-server)
+- [ ] it should install 2 binaries (cortex and cortex-server) [mac: binaries in `/usr/local/bin`]
 - [ ]  it should install with correct folder permissions
 - [ ]  it should install with folders: /engines /logs (no /models folder until model pull)
-
+- [ ] It should install with Docker image https://cortex.so/docs/installation/docker/
 
 ## Data/Folder structures
 - [ ] cortex.so models are stored in `cortex.so/model_name/variants/`, with .gguf and model.yml file
 - [ ] huggingface models are stored `huggingface.co/author/model_name` with .gguf and model.yml file
-- [ ] downloaded models are saved in cortex.db (view via SQL)
-- [ ] [to add] tests for copying models data folder & relative paths
-
+- [ ] downloaded models are saved in cortex.db with the right fields: `model`, `author_repo_id`, `branch_name`, `path_to_model_yaml` (view via SQL)
 
 ## Cortex Update
 - [ ] cortex -v should check output current version and check for updates
 - [ ] cortex update replaces the app, installer, uninstaller and binary file (without installing cortex.llamacpp)
-- [ ]  cortex update should update from ~3-5 versions ago to latest (+3 to 5 bump)
-- [ ]  cortex update should update from the previous version to latest (+1 bump)
-- [ ] cortex update should update from previous stable version to latest (stable checking)
-- [ ]  it should gracefully update when server is actively running
+- [ ]  `cortex update` should update from ~3-5 versions ago to latest (+3 to 5 bump)
+- [ ]  `cortex update` should update from the previous version to latest (+1 bump)
+- [ ] `cortex update -v 1.x.x-xxx` should update from the previous version to specified version
+- [ ] `cortex update` should update from previous stable version to latest
+- [ ] it should gracefully update when server is actively running
 
 ## Overall / App Shell
-- [ ] cortex returns helpful text in a timely* way
+- [ ] cortex returns helpful text in a timely* way (< 5s)
 - [ ] `cortex` or `cortex -h` displays help commands
-- [ ] CLI commands should start the API server, if not running [WIP `cortex pull`, `cortex engines install`]
+- [ ] CLI commands should start the API server, if not running [except 
 - [ ] it should correctly log to cortex-cli.log and cortex.log
 - [ ] There should be no stdout from inactive shell session
 
 ## Engines
 - [ ] llama.cpp should be installed by default
-- [ ]  it should run gguf models on llamacpp
-- [ ] it should install engines
-- [ ] it should list engines (Compatible, Ready, Not yet installed)
+- [ ] it should run gguf models on llamacpp
+- [ ] it should list engines
 - [ ] it should get engines
+- [ ] it should install engines (latest version if not specified)
+- [ ] it should install engines (with specified variant and version)
+- [ ] it should get default engine
+- [ ] it should set default engine (with specified variant/version)
+- [ ] it should load engine
+- [ ] it should unload engine
+- [ ] it should update engine (to latest version)
+- [ ] it should update engine (to specified version)
 - [ ] it should uninstall engines
 - [ ]  it should gracefully continue engine installation if interrupted halfway (partial download)
 - [ ]  it should gracefully handle when users try to CRUD incompatible engines (No variant found for xxx)
@@ -62,15 +68,17 @@ OS (select one)
 - [ ]  it should update engines versions [WIP, not tested]
 
 ## Server
-- [ ] `cortex start` should start server and output API documentation page
-- [ ] users can see API documentation page 
-- [ ]  `cortex stop` should stop server
-- [ ]  it should correctly log to cortex logs
+- [ ] `cortex start` should start server and output localhost URL & port number
+- [ ] users can access API Swagger documentation page  at localhost URL & port number
+- [ ] `cortex start` can be configured with parameters (port, [logLevel [WIP]](https://github.com/janhq/cortex.cpp/pull/1636)) https://cortex.so/docs/cli/start/
+- [ ]  it should correctly log to cortex logs (logs/cortex.log, logs/cortex-cli.log)
 - [ ]  `cortex ps` should return server status and running models (or no model loaded)
+- [ ]  `cortex stop` should stop server
 
 ## Model Pulling
 - [ ] Pulling a model should pull .gguf and model.yml file
-- [ ] Model download progress should appear (with accurate %, total time, download size, speed)
+- [ ] Model download progress should appear as download bars for each file
+- [ ] Model download progress should be accurate (%, total time, download size, speed)
 ### cortex.so
 - [ ]  it should pull by built in model_ID
 - [ ] pull by model_ID should recommend default variant at the top (set in HF model.yml)
@@ -85,24 +93,23 @@ OS (select one)
 
 ## Model Management
 - [ ]  it should list downloaded models
-- [ ] it should get info of a local model
-- [ ]  it should update models
+- [ ] it should get a local model
+- [ ]  it should update model parameters in model.yaml
 - [ ] it should delete a model
 - [ ]  it should import models with model_id and model_path
-- [ ] [To deprecate] it should alias models (deprecate once `cortex run` with regex is implemented)
 
 ## Model Running
 - [ ] `cortex run <cortexso model>` - if no local models detected, shows `pull` model menu
 - [ ] `cortex run` - if local model detected, runs the local model
-- [ ]  `cortex run` - if multiple local models detected, shows list of local models for users to select
+- [ ]  `cortex run` - if multiple local models detected, shows list of local models (from multiple model sources eg cortexso, HF authors) for users to select (via regex search)
 - [ ] `cortex run <invalid model id>` should return gracefully `Model not found!`
 - [ ]  run should autostart server
 - [ ] `cortex run <model>` starts interactive chat (by default)
 - [ ] `cortex run <model> -d` runs in detached mode
 - [ ] `cortex models start <model>`  
 - [ ] terminate StdIn or `exit()` should exit interactive chat
 
-## Hardware Detection / Acceleration [WIP]
+## Hardware Detection / Acceleration [WIP, no need to QA]
 - [ ]  it should auto offload max ngl
 - [ ]  it should correctly detect available GPUs
 - [ ]  it should gracefully detect missing dependencies/drivers
@@ -120,34 +127,47 @@ GPU Acceleration (e.g. CUDA11, CUDA12, Vulkan, sycl, etc)
 --
 # 2. API QA
 
-## Overall API
-- [ ] API page is updated at localhost:port endpoint (upon `cortex start`)
-- [ ] OpenAI compatibility for below
+## Checklist for each endpoint
+- [ ] Upon `cortex start`, API page is displayed at localhost:port endpoint
+- [ ] Endpoints should support the parameters stated in API reference (towards OpenAI Compatibility)
 - [ ] https://cortex.so/api-reference is updated
 
 ## Endpoints
 ### Chat Completions
 - [ ] POST `v1/chat/completions`
+- [ ] Cortex supports Function Calling #295
 
 ### Engines
-- [ ] GET `/v1/engines`
-- [ ] DELETE `/v1/engines/install/{name}`
-- [ ] POST `/v1/engines/install/{name}`
-- [ ] GET `/v1/engines/{name}`
-
-### Models
-- [ ] GET `/v1/models` lists models
-- [ ] POST `/v1/models/pull` starts download (websockets)
-- [ ] `websockets /events` emitted when model pull starts 
-- [ ] DELETE `/v1/models/pull` stops download (websockets)
-- [ ] `websockets /events` stopped when model pull stops
-- [ ] POST `/v1/models/start` starts model
-- [ ] POST `/v1/models/stop` stops model
-- [ ] DELETE `/v1/models/{id}` deletes model
-- [ ] GET `/v1/models/{id}` gets model
-- [ ] PATCH `/v1/models/{model}` updates model.yaml params
-
-----
-#### Test list for reference:
+- [ ] List engines: GET `/v1/engines`
+- [ ] Get engine: GET `/v1/engines/{name}`
+- [ ] Install engine: POST `/v1/engines/install/{name}`
+- [ ] Get default engine variant/version: GET `v1/engines/{name}/default`
+- [ ] Set default engine variant/version: POST `v1/engines/{name}/default`
+- [ ] Load engine: POST `v1/engines/{name}/load`
+- [ ] Unload engine: DELETE `v1/engines/{name}/load`
+- [ ] Update engine: POST `v1/engines/{name}/update`
+- [ ] uninstall engine: DELETE `/v1/engines/install/{name}`
+
+### Pulling Models
+- [ ] Pull model: POST `/v1/models/pull` starts download (websockets)
+- [ ] Pull model: `websockets /events` emitted  
+- [ ] Stop model download: DELETE `/v1/models/pull` (websockets)
+- [ ] Stop model download: `websockets /events` stopped
+- [ ] Import model: POST `v1/models/import`
+
+### Running Models
+- [ ] List models: GET `v1/models`
+- [ ] Start model: POST `/v1/models/start`
+- [ ] Stop model: POST `/v1/models/stop`
+- [ ] Get model: GET `/v1/models/{id}`
+- [ ] Delete model: DELETE `/v1/models/{id}`
+- [ ] Update model: PATCH `/v1/models/{model}` updates model.yaml params
+
+## Server
+- [ ] CORs [WIP]
+- [ ] health: GET `/healthz`
+- [ ] terminate server: DELETE `/processManager/destroy`
+--------
+Test list for reference:
 - #1357 e2e tests for APIs in CI
 - #1147, #1225 for starting QA list