Unable to build for Python 3.13 and CUDA 13

[bjhargrave]

The latest vLLM release (0.20.x) wheels on PyPI now use Python 3.13, PyTorch 2.11 and CUDA 13. When using cog, there is no base image for Python 3.13 and CUDA 13.

➜  docker pull r8.im/cog-base:cuda13.0-python3.13
Error response from daemon: manifest for r8.im/cog-base:cuda13.0-python3.13 not found: manifest unknown: Failed to fetch "cuda13.0-python3.13"

Trying to use Python 3.13+CUDA 12.8 (for which a cog base image does exist) causes a runtime failure as vLLM needs CUDA 13.

➜  COG_LEGACY_SCHEMA=1 cog-0.19.2 build --tag granite-vision-4.1-4b:1.0.0 --progress plain --separate-weights --use-cog-base-image

[...]

#15 naming to docker.io/library/granite-vision-4.1-4b:1.0.0 done
#15 DONE 22.8s
 ⚙  Validating model schema...

 ⚙  Traceback (most recent call last):
 ⚙    File "<frozen runpy>", line 198, in _run_module_as_main
 ⚙    File "<frozen runpy>", line 88, in _run_code
 ⚙    File "/root/.pyenv/versions/3.13.0/lib/python3.13/site-packages/cog/command/openapi_schema.py", line 110, in <module>
 ⚙      module_name, class_name = _load_module_from_ref(ref)
 ⚙                                ~~~~~~~~~~~~~~~~~~~~~^^^^^
 ⚙    File "/root/.pyenv/versions/3.13.0/lib/python3.13/site-packages/cog/command/openapi_schema.py", line 40, in _load_module_from_ref
 ⚙      spec.loader.exec_module(module)
 ⚙      ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
 ⚙    File "<frozen importlib._bootstrap_external>", line 1022, in exec_module
 ⚙    File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
 ⚙    File "/src/predict.py", line 35, in <module>
 ⚙      from vllm.config import ModelConfig, VllmConfig
 ⚙    File "/root/.pyenv/versions/3.13.0/lib/python3.13/site-packages/vllm/config/__init__.py", line 6, in <module>
 ⚙      from vllm.config.compilation import (
 ⚙      ...<4 lines>...
 ⚙      )
 ⚙    File "/root/.pyenv/versions/3.13.0/lib/python3.13/site-packages/vllm/config/compilation.py", line 22, in <module>
 ⚙      from vllm.platforms import current_platform
 ⚙    File "/root/.pyenv/versions/3.13.0/lib/python3.13/site-packages/vllm/platforms/__init__.py", line 278, in __getattr__
 ⚙      _current_platform = resolve_obj_by_qualname(platform_cls_qualname)()
 ⚙                          ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
 ⚙    File "/root/.pyenv/versions/3.13.0/lib/python3.13/site-packages/vllm/utils/import_utils.py", line 109, in resolve_obj_by_qualname
 ⚙      module = importlib.import_module(module_name)
 ⚙    File "/root/.pyenv/versions/3.13.0/lib/python3.13/importlib/__init__.py", line 88, in import_module
 ⚙      return _bootstrap._gcd_import(name[level:], package, level)
 ⚙             ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 ⚙    File "/root/.pyenv/versions/3.13.0/lib/python3.13/site-packages/vllm/platforms/cuda.py", line 21, in <module>
 ⚙      import vllm._C  # noqa
 ⚙      ^^^^^^^^^^^^^^
 ⚙  ImportError: libcudart.so.13: cannot open shared object file: No such file or directory

 ✗ image build failed: Failed to get type signature: container exited with status 1

Next I tried --use-cuda-base-image true which leads to errors with uv pip install failing trying to install the cog package.

➜  COG_LEGACY_SCHEMA=1 cog-0.19.2 build --tag granite-vision-4.1-4b:1.0.0 --progress plain --separate-weights --use-cuda-base-image true
 ⚙  Building Docker image from environment in cog.yaml as granite-vision-4.1-4b:1.0.0...

 ⚠ Could not find a suitable base image, continuing without base image support (unsupported base image configuration: CUDA: 13.0 / Python: 3.13 /
 ⚠ Torch: (none)).
 ⚙  Weights unchanged, skip rebuilding and use cached image...
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 1.93kB done
#1 DONE 0.0s

#2 resolve image config for docker-image://docker.io/docker/dockerfile:1.4
#2 DONE 0.1s

#3 docker-image://docker.io/docker/dockerfile:1.4@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc
#3 CACHED

#4 [internal] load .dockerignore
#4 transferring context: 776B done
#4 DONE 0.0s

#5 [internal] load metadata for ghcr.io/astral-sh/uv:0.9.26
#5 DONE 0.1s

#6 [internal] load metadata for docker.io/nvidia/cuda:13.0.2-cudnn-devel-ubuntu24.04
#6 DONE 0.2s

#7 [stage-1  1/10] FROM docker.io/nvidia/cuda:13.0.2-cudnn-devel-ubuntu24.04@sha256:e071e85c52ad91fc9ea24158ff5330876b2d1a5c4ac83ccc6066976835873c01
#7 DONE 0.0s

#8 FROM ghcr.io/astral-sh/uv:0.9.26@sha256:9a23023be68b2ed09750ae636228e903a54a05ea56ed03a934d00fe9fbeded4b
#8 DONE 0.0s

#9 [stage-1  4/10] COPY --from=ghcr.io/astral-sh/uv:0.9.26 /uv /uvx /usr/local/bin/
#9 CACHED

[...] 

#13 [stage-1  5/10] RUN uv python install 3.13 && 	ln -sf $(uv python find 3.13) /usr/bin/python3
#13 0.296 Downloading cpython-3.13.11-linux-x86_64-gnu (download) (33.4MiB)
#13 0.908  Downloaded cpython-3.13.11-linux-x86_64-gnu (download)
#13 0.948 Installed Python 3.13.11 in 781ms
#13 0.948  + cpython-3.13.11-linux-x86_64-gnu (python3.13)
#13 0.949 warning: `/root/.local/bin` is not on your PATH. To use installed Python executables, add the directory to your PATH.
#13 DONE 1.3s

#14 [stage-1  6/10] RUN --mount=type=cache,target=/root/.cache/uv uv pip install --no-cache cog==0.19.2
#14 0.459 Using Python 3.13.11 environment at: /root/.local/share/uv/python/cpython-3.13.11-linux-x86_64-gnu
#14 0.460 error: The interpreter at /root/.local/share/uv/python/cpython-3.13.11-linux-x86_64-gnu is externally managed, and indicates the following:
#14 0.460
#14 0.460   This Python installation is managed by uv and should not be modified.
#14 0.460
#14 0.460 hint: Virtual environments were not considered due to the `--system` flag
#14 ERROR: process "/bin/sh -c uv pip install --no-cache cog==0.19.2" did not complete successfully: exit code: 2
------
 > [stage-1  6/10] RUN --mount=type=cache,target=/root/.cache/uv uv pip install --no-cache cog==0.19.2:
0.459 Using Python 3.13.11 environment at: /root/.local/share/uv/python/cpython-3.13.11-linux-x86_64-gnu
0.460 error: The interpreter at /root/.local/share/uv/python/cpython-3.13.11-linux-x86_64-gnu is externally managed, and indicates the following:
0.460
0.460   This Python installation is managed by uv and should not be modified.
0.460
0.460 hint: Virtual environments were not considered due to the `--system` flag
------
 ✗ image build failed: Failed to build runner Docker image: Failed to build Docker image: failed to solve: process "/bin/sh -c uv pip install
 ✗ --no-cache cog==0.19.2" did not complete successfully: exit code: 2

Next I tried --use-cuda-base-image false which builds but fails at runtime since vLLM excepts the CUDA tools (e.g. nvcc) are not in the python:3.13-slim image. I don't want to deal with and manage the RUN steps to install the necessary CUDA tools.

Failed to find C compiler. Please specify via CC environment variable or set triton.knobs.build.impl.

So I am stuck. I need to use Python 3.13 and CUDA 13 but am unable to find a cog base image or use the CUDA base image. Please provide a r8.im/cog-base:cuda13.0-python3.13 cog base image or fix cog to enable uv pip install with the CUDA image.

Thanks.

[bjhargrave]

cc: @markphelps @michaeldwan Here is the next issue I have encountered trying to build for the new granite-vision-4.1-4b model.

https://github.com/bjhargrave/cog-models/tree/granite-vision-4.1-4b/ibm-granite/granite-vision-4.1-4b

[michaeldwan]

Thanks for reporting. We'll get a base image up asap.

[markphelps]

@bjhargrave we've published r8.im/cog-base:cuda13.0-python3.13 and similar python 13.3 base images

[bjhargrave]

@markphelps, I was able to test using the r8.im/cog-base:cuda13.0-python3.13 image but I am now stuck on a uv error with the installed python interpreter. The cog base images seems to use a uv managed python interpreter and this conflicts with using uv pip install without a virtualenv.

➜  COG_LEGACY_SCHEMA=1 cog-0.19.2 build --progress plain --separate-weights --use-cog-base-image
 ⚙  Building Docker image from environment in cog.yaml as r8.im/ibm-granite/granite-vision-4.1-4b:1.0.0...

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 138B done
#1 DONE 0.0s

#2 resolve image config for docker-image://docker.io/docker/dockerfile:1.4
#2 DONE 0.1s

#3 docker-image://docker.io/docker/dockerfile:1.4@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc
#3 CACHED

#4 [internal] load .dockerignore
#4 transferring context:
#4 transferring context: 776B done
#4 DONE 0.0s

#5 exporting to image
#5 preparing layers for inline cache done
#5 writing image sha256:71de1148337f4d1845be01eb4caf15d78e4eb15a1ab96030809826698a5b7e30 done
#5 naming to r8.im/ibm-granite/granite-vision-4.1-4b:1.0.0-weights done
#5 DONE 0.0s
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 873B done
#1 DONE 0.0s

#2 resolve image config for docker-image://docker.io/docker/dockerfile:1.4
#2 DONE 0.1s

#3 docker-image://docker.io/docker/dockerfile:1.4@sha256:9ba7531bd80fb0a858632727cf7a112fbfd19b17e94c4e84ced81e24ef1a0dbc
#3 CACHED

#4 [internal] load .dockerignore
#4 transferring context: 776B done
#4 DONE 0.0s

#5 [internal] load metadata for r8.im/cog-base:cuda13.0-python3.13
#5 DONE 0.0s

#6 [internal] load metadata for ghcr.io/astral-sh/uv:0.9.26
#6 DONE 3.1s

#7 [stage-1 1/6] FROM r8.im/cog-base:cuda13.0-python3.13
#7 CACHED

#8 [internal] load build context
#8 ...

#9 FROM ghcr.io/astral-sh/uv:0.9.26@sha256:9a23023be68b2ed09750ae636228e903a54a05ea56ed03a934d00fe9fbeded4b
#9 resolve ghcr.io/astral-sh/uv:0.9.26@sha256:9a23023be68b2ed09750ae636228e903a54a05ea56ed03a934d00fe9fbeded4b 0.0s done
#9 sha256:08a7428e3daeb4ff634fe06d3d9aec278579e88f770b5d141e5a408cb998f40a 669B / 669B done
#9 sha256:02338ef4d522b073c9469e1656cfa160407cb7160023300bb8622c9757778b13 1.30kB / 1.30kB done
#9 sha256:5b10ebfb4e369f07b34982c85caaab98838a61ebb54e54946271f61d7cf1f8a6 23.02MB / 23.02MB 0.2s done
#9 sha256:b3d2c6abdae87c43a56d2830b783d600de9274bb8791534edac6e02d23667aa3 98B / 98B 0.1s done
#9 sha256:9a23023be68b2ed09750ae636228e903a54a05ea56ed03a934d00fe9fbeded4b 2.19kB / 2.19kB done
#9 extracting sha256:5b10ebfb4e369f07b34982c85caaab98838a61ebb54e54946271f61d7cf1f8a6 0.2s done
#9 extracting sha256:b3d2c6abdae87c43a56d2830b783d600de9274bb8791534edac6e02d23667aa3 done
#9 DONE 0.4s

#8 [internal] load build context
#8 ...

#10 [stage-1 2/6] COPY --from=ghcr.io/astral-sh/uv:0.9.26 /uv /uvx /usr/local/bin/
#10 DONE 0.1s

#8 [internal] load build context
#8 transferring context: 437.99MB 1.5s done
#8 CANCELED

#11 [stage-1 3/6] RUN --mount=type=cache,target=/root/.cache/uv uv pip install --no-cache cog==0.19.2
#11 0.535 Using Python 3.13.11 environment at: /root/.local/share/uv/python/cpython-3.13.11-linux-x86_64-gnu
#11 0.538 error: The interpreter at /root/.local/share/uv/python/cpython-3.13.11-linux-x86_64-gnu is externally managed, and indicates the following:
#11 0.538
#11 0.538   This Python installation is managed by uv and should not be modified.
#11 0.538
#11 0.538 hint: Virtual environments were not considered due to the `--system` flag
#11 ERROR: process "/bin/sh -c uv pip install --no-cache cog==0.19.2" did not complete successfully: exit code: 2
------
 > [stage-1 3/6] RUN --mount=type=cache,target=/root/.cache/uv uv pip install --no-cache cog==0.19.2:
0.535 Using Python 3.13.11 environment at: /root/.local/share/uv/python/cpython-3.13.11-linux-x86_64-gnu
0.538 error: The interpreter at /root/.local/share/uv/python/cpython-3.13.11-linux-x86_64-gnu is externally managed, and indicates the following:
0.538
0.538   This Python installation is managed by uv and should not be modified.
0.538
0.538 hint: Virtual environments were not considered due to the `--system` flag
------
 ✗ image build failed: Failed to build runner Docker image: Failed to build Docker image: failed to solve: process "/bin/sh -c uv pip install
 ✗ --no-cache cog==0.19.2" did not complete successfully: exit code: 2

I am also not sure why the weights Dockerfile does not copy the weights folder into the image?

➜  COG_LEGACY_SCHEMA=1 cog-0.19.2 debug --separate-weights --use-cog-base-image
=== Weights Dockerfile contents:
#syntax=docker/dockerfile:1.4
FROM scratch

===

=== Runner Dockerfile contents:
#syntax=docker/dockerfile:1.4
FROM cog-granite-vision-41-4b-weights AS weights
FROM r8.im/cog-base:cuda13.0-python3.13
COPY --from=ghcr.io/astral-sh/uv:0.9.26 /uv /uvx /usr/local/bin/
ENV UV_SYSTEM_PYTHON=true
ENV CFLAGS="-O3 -funroll-loops -fno-strict-aliasing -flto -S"
RUN --mount=type=cache,target=/root/.cache/uv uv pip install --no-cache cog==0.19.2
ENV CFLAGS=
RUN --mount=type=cache,target=/root/.cache/uv --mount=type=bind,source=requirements.txt,target=/tmp/requirements.txt uv pip install --cache-dir /root/.cache/uv --requirements /tmp/requirements.txt --torch-backend=cu130 --link-mode=copy
WORKDIR /src
EXPOSE 5000
ENV COG_PREDICT_TYPE_STUB="predict.py:Predictor"
ENV COG_MAX_CONCURRENCY=4
CMD ["python", "-m", "cog.server.http"]
COPY . /src
===

=== DockerIgnore contents:
# generated by replicate/cog
__pycache__
*.pyc
*.pyo
*.pyd
.Python
env
pip-log.txt
pip-delete-this-directory.txt
.tox
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.log
.git
.mypy_cache
.pytest_cache
.hypothesis
===