CUDA capability

[billila]

Hi @vqdang !
I am encountering an issue while running run_inference.py.
I receive the following error message:

|2024-11-18|18:34:01.092| [INFO] .... Detect #GPUS: 1
/home/x-ibillato/.conda/envs/2024.02-py311/hovernet/lib/python3.6/site-packages/torch/cuda/__init__.py:125: UserWarning: 
NVIDIA A100-SXM4-40GB with CUDA capability sm_80 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70 sm_75.
If you want to use the NVIDIA A100-SXM4-40GB GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/
warnings.warn(incompatible_device_warn.format(device_name, capability, " ".join(arch_list), device_name))
|2024-11-18|18:34:03.617| [INFO] ................ Process: TCGA-02-0001-01Z-00-DX1.83fce43e-42ac-4dcd-b156-2908e75f2e47
|2024-11-18|18:35:20.704| [INFO] ................ WARNING: No mask found, generating mask via thresholding at 1.25x!
|2024-11-18|18:35:43.621| [INFO] ........ Preparing Input Output Placement: 100.00288363406435

Process Chunk 3/80:   0%|                                 | 0/3 [00:00<?, ?it/s]|2024-11-18|18:35:50.790| [ERROR] Crash
Traceback (most recent call last):
  File "/home/x-ibillato/hover_net/infer/wsi.py", line 748, in process_wsi_list
    self.process_single_file(wsi_path, msk_path, self.output_dir)
  File "/home/x-ibillato/hover_net/infer/wsi.py", line 550, in process_single_file
    self.__get_raw_prediction(chunk_info_list, patch_info_list)
  File "/home/x-ibillato/hover_net/infer/wsi.py", line 374, in __get_raw_prediction
    chunk_patch_info_list[:, 0, 0], pbar_desc
  File "/home/x-ibillato/hover_net/infer/wsi.py", line 289, in __run_model
    sample_output_list = self.run_step(sample_data_list)
  File "/home/x-ibillato/hover_net/infer/base.py", line 74, in <lambda>
    self.run_step = lambda input_batch: run_step(input_batch, net)
  File "/home/x-ibillato/hover_net/models/hovernet/run_desc.py", line 176, in infer_step
    patch_imgs_gpu = patch_imgs.to("cuda").type(torch.float32)  # to NCHW
RuntimeError: CUDA error: no kernel image is available for execution on the device

Process Chunk 3/80:   0%|                                 | 0/3 [00:06<?, ?it/s]

I followed the instructions in the set-up-enviroment guide to set up my conda environment.
The NVIDIA A100-SXM4-40GB GPU seems incompatible with my current PyTorch installation, which only supports CUDA capabilities up to sm_75.

thanks for your help!

[simongraham]

The repo supports PyTorch version 1.6 with CUDA 10.2. We can't guarantee that the code will work for later torch / CUDA versions.

[KondratievaOlesya]

I was able to run inference on an A100 GPU (CUDA capability sm_80) by creating a custom conda environment. Since the A100 requires CUDA 11.0 or higher, and imgaug was not compatible with the latest NumPy (2.x), I had to manually install dependencies and downgrade numpy to avoid breaking changes.

Conda enviroment

name: hovernet_a100
channels:
  - defaults
  - conda-forge
dependencies:
  - python=3.9
  - openslide
  - pip
  - pip:
      - docopt==0.6.2
      - future==0.18.2
      - imgaug==0.4.0
      - matplotlib==3.3.0
      - opencv-python
      - pandas
      - scikit-image
      - psutil
      - scikit-learn
      - scipy
      - tensorboard
      - tensorboardx
      - termcolor
      - tqdm
      - torch
      - torchvision
      - openslide-python
      - "numpy<2.0"

Manual environment setup

This is how I did it

Create a clean conda environment

conda create -n hovernet_a100 python=3.9
conda activate hovernet_a100

Install system-level dependency

conda install openslide

Install required python packages

pip install docopt==0.6.2
pip install future==0.18.2
pip install imgaug==0.4.0
pip install matplotlib==3.3.0
pip install opencv-python
pip install pandas
pip install scikit-image
pip install psutil
pip install scikit-learn
pip install scipy
pip install tensorboard
pip install tensorboardx
pip install termcolor
pip install tqdm
pip install torch       # Automatically installs the compatible CUDA version
pip install torchvision
pip install openslide-python

IMPORTANT: Downgrade numpy to avoid incompatibility with imgaug

pip install "numpy<2.0"