Add CUDA version compatibility check #1412

Andy-Jost · 2025-12-18T19:06:15Z

Summary

Adds a version compatibility check that warns users when cuda-bindings was compiled against a newer CUDA major version than the installed driver supports.

Changes

cuda-bindings

Added check_cuda_version_compatibility() function in cuda/bindings/utils/_version_check.py
Compares compile-time CUDA_VERSION vs runtime cuDriverGetVersion()
Exported from cuda.bindings.utils
Added comprehensive unit tests in tests/test_version_check.py

cuda-core

Device.__new__ calls check_cuda_version_compatibility() after cuInit succeeds
Imports the function directly from cuda.bindings.utils

Rationale

When cuda-bindings is built against CUDA 13 headers but the user's driver only supports CUDA 12, many features will silently fail or behave unexpectedly. This check provides early, clear feedback:

UserWarning: cuda-bindings was built against CUDA 13.0, but the installed driver 
only supports CUDA 12.8. Some features may not work correctly. Consider updating 
your NVIDIA driver. Set CUDA_PYTHON_DISABLE_VERSION_CHECK=1 to suppress this warning.

Design

Provided by cuda-bindings: The version check implementation lives in cuda.bindings.utils since it checks cuda-bindings' compile-time version
Invoked by cuda-core: Called when Device first triggers CUDA initialization
Runs once: Uses a module-level flag to ensure the check runs only once per process
Non-blocking: Warning only, does not prevent operation
Suppressible: Set CUDA_PYTHON_DISABLE_VERSION_CHECK=1 to disable
Major version only: No warning for minor version differences (handled by graceful degradation per PR Handle unsupported device attributes gracefully #1409)

Future Work

We could not find a suitable place to invoke the version check automatically within cuda-bindings itself (e.g., hooking into cuInit), so the check is currently triggered by cuda-core. This may be revisited in the future.

Test Coverage

7 tests in cuda-bindings covering:

No warning when driver is newer
No warning when same major version
Warning when compile major > driver major
Warning only issued once per process
Suppression via environment variable
Silent handling of driver errors
Silent handling of missing CUDA_VERSION attribute

Related Work

Complements PR Handle unsupported device attributes gracefully #1409 which handles graceful degradation for minor version mismatches

copy-pr-bot · 2025-12-18T19:06:18Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Andy-Jost · 2025-12-18T19:06:28Z

/ok to test 7ce325c

Andy-Jost · 2025-12-18T19:08:17Z

/ok to test 1962e35

github-actions · 2025-12-18T19:20:22Z

Doc Preview CI
🚀 View preview at https://nvidia.github.io/cuda-python/pr-preview/pr-1412/
https://nvidia.github.io/cuda-python/pr-preview/pr-1412/cuda-core/
https://nvidia.github.io/cuda-python/pr-preview/pr-1412/cuda-bindings/
https://nvidia.github.io/cuda-python/pr-preview/pr-1412/cuda-pathfinder/
Preview will be ready when the GitHub Pages deployment is complete.

cuda_core/cuda/core/_utils/cuda_utils.pyx

cuda_bindings/docs/source/environment_variables.rst

cuda_core/cuda/core/_utils/cuda_utils.pyx

Andy-Jost · 2025-12-19T00:10:25Z

/ok to test 73e5a43

Andy-Jost · 2025-12-19T00:48:39Z

/ok to test ddd92bd

cuda_bindings/cuda/bindings/utils/_version_check.py

rparolin

We should issue a warning to the user if we can't fetch the driver version.

cuda_bindings/cuda/bindings/utils/_version_check.py

Add warn_if_cuda_major_version_mismatch() to cuda-bindings that warns when cuda-bindings was compiled for a newer CUDA major version than the installed driver supports. Called by cuda.core on first Device access.

# Conflicts: # cuda_core/pixi.lock

Andy-Jost · 2026-01-07T21:11:36Z

/ok to test 62dfcca

Import warn_if_cuda_major_version_mismatch locally in Device.__new__ after cuInit, using try/except/else pattern instead of module-level import with lambda fallback.

Andy-Jost · 2026-01-08T18:00:34Z

/ok to test b2083ed

Extract Device.__new__ logic into cdef helper functions: - Device_ensure_cuda_initialized(): cuInit + version check - Device_resolve_device_id(): resolve None to current device or 0 - Device_ensure_tls_devices(): create thread-local singletons Reduces Device.__new__ from ~60 lines to ~12 lines. Helpers placed after Device class following memory module pattern.

Andy-Jost · 2026-01-08T18:19:03Z

/ok to test fdb3a7e

rparolin · 2026-01-09T21:08:17Z

cuda_bindings/tests/test_version_check.py

+
+    def setup_method(self):
+        """Reset the version compatibility check flag before each test."""
+        _version_check._major_version_compatibility_checked = False


This is effectively going to reset if the version was checked already and after the unit tests executes the check could potentially execute again. Shouldn't we cache the current state and before setting it to false for the unit tests and then restore to its previous value on unit test teardown?

Someone with a dodgy build might get multiple warnings during testing. Not sure it's worth fixing.

Andy-Jost · 2026-01-09T21:30:52Z

/ok to test acadb31

Andy-Jost added this to the cuda.core beta 11 milestone Dec 18, 2025

Andy-Jost added P0 High priority - Must do! feature New feature or request cuda.core Everything related to the cuda.core module labels Dec 18, 2025

Andy-Jost self-assigned this Dec 18, 2025

Andy-Jost requested review from leofang, rparolin and rwgk December 18, 2025 19:06

Andy-Jost force-pushed the runtime-version-check branch from 7ce325c to 1962e35 Compare December 18, 2025 19:08

mdboom approved these changes Dec 18, 2025

View reviewed changes

Andy-Jost mentioned this pull request Dec 18, 2025

fix: derive CUDA major version from headers for build #1395

Merged

2 tasks

leofang requested changes Dec 18, 2025

View reviewed changes

cuda_core/cuda/core/_utils/cuda_utils.pyx Outdated Show resolved Hide resolved

leofang reviewed Dec 18, 2025

View reviewed changes

cuda_bindings/docs/source/environment_variables.rst Outdated Show resolved Hide resolved

leofang reviewed Dec 18, 2025

View reviewed changes

cuda_core/cuda/core/_utils/cuda_utils.pyx Outdated Show resolved Hide resolved

Andy-Jost force-pushed the runtime-version-check branch 2 times, most recently from 8a0d248 to 73e5a43 Compare December 19, 2025 00:09

Andy-Jost force-pushed the runtime-version-check branch from 73e5a43 to ddd92bd Compare December 19, 2025 00:47

leofang self-requested a review December 19, 2025 01:22

rparolin reviewed Dec 19, 2025

View reviewed changes

cuda_bindings/cuda/bindings/utils/_version_check.py Outdated Show resolved Hide resolved

rparolin reviewed Dec 19, 2025

View reviewed changes

cuda_bindings/cuda/bindings/utils/_version_check.py Outdated Show resolved Hide resolved

rparolin requested changes Dec 20, 2025

View reviewed changes

rwgk reviewed Dec 26, 2025

View reviewed changes

Andy-Jost added 2 commits January 7, 2026 13:09

Add CUDA major version compatibility check

d999f40

Add warn_if_cuda_major_version_mismatch() to cuda-bindings that warns when cuda-bindings was compiled for a newer CUDA major version than the installed driver supports. Called by cuda.core on first Device access.

Merge remote-tracking branch 'origin/main' into runtime-version-check

62dfcca

# Conflicts: # cuda_core/pixi.lock

Andy-Jost force-pushed the runtime-version-check branch from ddd92bd to 62dfcca Compare January 7, 2026 21:10

Andy-Jost requested a review from rparolin January 8, 2026 17:48

Andy-Jost added 2 commits January 8, 2026 09:59

Move version check import to local scope

73611ed

Import warn_if_cuda_major_version_mismatch locally in Device.__new__ after cuInit, using try/except/else pattern instead of module-level import with lambda fallback.

Merge remote-tracking branch 'origin/main' into runtime-version-check

b2083ed

Andy-Jost force-pushed the runtime-version-check branch from 07bf4a4 to b2083ed Compare January 8, 2026 18:00

Andy-Jost force-pushed the runtime-version-check branch from a92ed3b to fdb3a7e Compare January 8, 2026 18:18

rparolin reviewed Jan 9, 2026

View reviewed changes

rparolin approved these changes Jan 9, 2026

View reviewed changes

Merge branch 'main' into runtime-version-check

acadb31

Add CUDA version compatibility check #1412

Are you sure you want to change the base?

Add CUDA version compatibility check #1412

Conversation

Andy-Jost commented Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

cuda-bindings

cuda-core

Rationale

Design

Future Work

Test Coverage

Related Work

Uh oh!

copy-pr-bot bot commented Dec 18, 2025

Uh oh!

Andy-Jost commented Dec 18, 2025

Uh oh!

Andy-Jost commented Dec 18, 2025

Uh oh!

github-actions bot commented Dec 18, 2025

Preview will be ready when the GitHub Pages deployment is complete.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Andy-Jost commented Dec 19, 2025

Uh oh!

Andy-Jost commented Dec 19, 2025

Uh oh!

Uh oh!

Uh oh!

rparolin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Andy-Jost commented Jan 7, 2026

Uh oh!

Andy-Jost commented Jan 8, 2026

Uh oh!

Andy-Jost commented Jan 8, 2026

Uh oh!

rparolin Jan 9, 2026

Choose a reason for hiding this comment

Uh oh!

Andy-Jost Jan 9, 2026

Choose a reason for hiding this comment

Uh oh!

Andy-Jost commented Jan 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Andy-Jost commented Dec 18, 2025 •

edited

Loading