diff --git a/docker-compose.yml b/docker-compose.yml index 1764beca..18724033 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -49,8 +49,8 @@ services: # can replace this with naptha image once it's published to hub, then we just # need to use the .env file UNLESS config.py is modified container_name: node-app - build: - dockerfile: Dockerfile-node-dev + image: napthaai/node:latest + env_file: - ./.env volumes: diff --git a/docs/advanced.md b/docs/advanced.md index 30b03819..076ecd66 100644 --- a/docs/advanced.md +++ b/docs/advanced.md @@ -65,7 +65,16 @@ Note that the `RMQ_*` variables will be passed in to create the default RMQ user ## Building the Docker Image -The Naptha node uses a cross-platform Dockerfile intended for use with `docker buildx`. This is automatically used by Naptha's CI/CD, so when you pull the image, the right architecture will be used. To build the image with `docker buildx` yourself, you must first install Docker on your system. Once you have done that, complete the following steps to enable the `buildx` plugin: +The Naptha node uses a platform-agnostic Dockerfile intended for use with `docker buildx`. + +This is automatically used by Naptha's CI/CD, so when you pull the image from `napthaai/node:latest`, the right architecture will be available for your device, and will be pulled automatically. However, legacy `docker build` commands are automatically supported: + +```shell +docker build -t account/repository:tag -f Dockerfile-node . +``` + + +To build the image with `docker buildx` yourself, you must first install Docker on your system. Once you have done that, complete the following steps to enable the `buildx` plugin: ```shell @@ -82,28 +91,33 @@ push the manifests up to your dockerhub / other container repo, since the local does not support manifest lists for multi-platform builds. In that case, you need to specify a full container tag of `account/repository:tag` +To build the image for the architecture of your machine, run: + +```shell +docker buildx build \ + -t account/repository:tag \ + -f Dockerfile-node \ + . --load +``` + +Or, you can use the `--platform` flag to specify one or more specific architectures to build for. +This can be useful for example if you want to build the node on an ARM machine (e.g. a M-series Macbook) +and then run it on a remote server with a different architecture. + ```shell # for ARM CPUs only: docker buildx build \ - --platform linux/arm64 \ + --platform linux/arm64 # you can specify one or more target -t example-docker-tag \ - -f buildx.Dockerfile \ - --load . + -f Dockerfile-node \ + . --load # for multiple target platforms: docker buildx build \ --platform linux/arm64,linux/amd64 \ -t account/repository:tag \ - -f buildx.Dockerfile \ - --push . - + -f Dockerfile-node \ + . --load +``` -# for GPU-accelerated inference with ollama or vLLM; use an nvidia/cuda base image. replace 12.4.1 with your CUDA version; -# see https://hub.docker.com/r/nvidia/cuda/tags for a list of supported images. -docker buildx build \ - --platform linux/arm64 \ - --build-arg BASE_IMAGE=nvidia/cuda:12.4.1-devel-ubuntu22.04 \ - -t yourdockerprofile/yourrepo:yourtag \ - -f buildx.Dockerfile \ - --load . -``` \ No newline at end of file +Remember that with `docker buildx`, images are not automatically added to your build cache. You can use `--load` to load the image into your local docker cache, or use `--push` to push the image to a remote container repository. \ No newline at end of file diff --git a/docs/docker.md b/docs/docker.md index 2f459b9b..9ae15aec 100644 --- a/docs/docker.md +++ b/docs/docker.md @@ -38,20 +38,28 @@ The images will be pulled from `napthaai/node:latest` on Docker hub. It will not The docker compose runs in detached mode, so you can use `docker compose logs -f` to see the logs of the node. You can check the logs of individual services using: ```bash -docker logs -f +docker logs -f ``` -where `` is the name of the service you want to check the logs of e.g. `node-app`, `litellm`, `node-ollama`, `Hermes-3-Llama-3.1-8B`, `node-pgvector`, etc. You can run `docker ps -a` to see the names of all the services running in docker compose. +where `` is the id of the service you want to check the logs of. You can get a list of all currently-running containers and their IDs by running `docker ps`. +Some containers in the compose configuration have human-readable `container_name` fields set, e.g. `node-app`, `litellm`, `node-ollama`, `Hermes-3-Llama-3.1-8B`, `node-pgvector`, etc. in which can you can run + +```bash +docker logs -f +``` ### Accessing a Service's Container -To explore a service's filesystem or run commands inside a container, you can use: +To run commands inside a container using an interactive shell, e.g. to explore the filesystem, you can use: ```bash -docker exec -it /bin/bash +docker exec -it /bin/bash + +# or, for some containers that don't have bash: +docker exec -it /bin/sh ``` -and `exit` to exit the container. +Then, you can run `exit` to exit the shell inside the container when you are finished. This will not stop the container, it will just exit the shell. ## Stopping the Node @@ -61,5 +69,9 @@ To stop the node, you can use: bash docker-ctl.sh down ``` +This script is _auto-generated_ by `launch.sh` and is used to stop the node and remove the containers. It will not be present in the repository until you start the node with `launch.sh`. + +Note that by default, this will _not_ remove persistant volumes created by certain containers, e.g. the postgres database and the rabbitMQ server. If you want to remove these volumes, you can run `docker volume ls` and `docker volume rm` to find, list, and remove the volume(s) that you want to remove. + diff --git a/launch.sh b/launch.sh index f0ca8097..83fac626 100755 --- a/launch.sh +++ b/launch.sh @@ -2077,7 +2077,7 @@ launch_docker() { #!/bin/bash case "\$1" in "down") - env \$(cat .env | grep -v '^#' | xargs) $GPU_ASSIGNMENTS docker compose -f docker-compose.yml $COMPOSE_FILES down -v + env \$(cat .env | grep -v '^#' | xargs) $GPU_ASSIGNMENTS docker compose -f docker-compose.yml $COMPOSE_FILES down ;; "logs") env \$(cat .env | grep -v '^#' | xargs) $GPU_ASSIGNMENTS docker compose -f docker-compose.yml $COMPOSE_FILES logs -f @@ -2096,7 +2096,7 @@ EOF #!/bin/bash case "\$1" in "down") - docker compose -f docker-compose.yml $COMPOSE_FILES down -v + docker compose -f docker-compose.yml $COMPOSE_FILES down ;; "logs") docker compose -f docker-compose.yml $COMPOSE_FILES logs -f diff --git a/node/compose-files/vllm-models/DeepHermes-3-Llama-3-8B-Preview.yml b/node/compose-files/vllm-models/DeepHermes-3-Llama-3-8B-Preview.yml new file mode 100644 index 00000000..cf886f62 --- /dev/null +++ b/node/compose-files/vllm-models/DeepHermes-3-Llama-3-8B-Preview.yml @@ -0,0 +1,33 @@ +services: + DeepHermes-3-Llama-3-8B-Preview: + image: vllm/vllm-openai:latest + entrypoint: [ + "vllm", "serve", "NousResearch/DeepHermes-3-Llama-3-8B-Preview", + "--enable-prefix-caching", "--enable-chunked-prefill", + "--gpu-memory-utilization", "0.98", + "--max-model-len", "131072", + "--enable-auto-tool-choice", "--tool-call-parser", "hermes" + ] + environment: + HUGGING_FACE_HUB_TOKEN: ${HUGGINGFACE_TOKEN:?error} + volumes: + - type: bind + source: ${HF_HOME:?error} + target: /root/.cache/huggingface + - type: bind + source: ./node/inference/configs + target: /usr/app # configs like chat templates, vllm configs, tool parsers + ipc: host + deploy: + resources: + reservations: + devices: + - driver: "nvidia" + capabilities: [ "gpu" ] + device_ids: ["${GPU_ID_deephermes_3_llama_3_8b_preview:?error}"] + networks: + - naptha-network + +networks: + naptha-network: + external: true diff --git a/node/inference/litellm/generate_litellm_config.py b/node/inference/litellm/generate_litellm_config.py index 7081569e..861e2562 100644 --- a/node/inference/litellm/generate_litellm_config.py +++ b/node/inference/litellm/generate_litellm_config.py @@ -27,7 +27,8 @@ "deepseek-ai/DeepSeek-R1-Distill-Qwen-32B": 2, "microsoft/phi-4": 1, "mistralai/Mistral-Small-24B-Instruct-2501": 2, - "Qwen/QwQ-32B-Preview": 2 + "Qwen/QwQ-32B-Preview": 2, + "NousResearch/DeepHermes-3-Llama-3-8B-Preview": 1 } if VLLM_MODELS: diff --git a/node/storage/hub/hub.py b/node/storage/hub/hub.py index a360b657..1d9304a3 100644 --- a/node/storage/hub/hub.py +++ b/node/storage/hub/hub.py @@ -22,6 +22,7 @@ def __init__(self, *args, **kwargs): self.hub_url = LOCAL_HUB_URL else: self.hub_url = PUBLIC_HUB_URL + self.ns = os.getenv("HUB_DB_SURREAL_NS") self.db = os.getenv("HUB_DB_SURREAL_NAME")