[hipFile] Python API using Cython bindings by systems-assistant[bot] · Pull Request #4865 · ROCm/rocm-systems

systems-assistant · 2026-04-09T16:25:27Z

This is an initial release for Python bindings for hipFile. This is still in its early stages and will likely change substantially between now and what could be called an "official" release. The generated Cython bindings should be considered temporary as we work with another team for a long-term solution, but they at least give a base to build off of for the higher-level Python API.

Addresses ROCm/hipFile#201

Test Plan

Can we load the C hipFile library from Python and run select functions?
Can we run IO?

Test Result

It works :)

AIHIPFILE-140

🔁 Imported from ROCm/hipFile#248
🧑‍💻 Originally authored by @riley-dixon

Also move _hipfile driver functions out of public __init__.py

This exception will be thrown by the Python hipFile module rather than returning an error struct. The hipFile C error macros are unused since they aren't very Pythonic in the first place. If the error is a hipFile error (even if it is a wrapped HIP error), we raise HipFileException. If it is a different error, a different Exception will be raised. We also won't be exposing the error struct/tuple to Python users, though it can be accessed via the HipFileException when raised.

This module will contain the functions related to querying driver/library configuration/properties in a Python friendly manner.

A fairly rudimentary interface for opening/closing files with hipFile.

A simple wrapper. Still need to figure out how exactly a hipFile Buffer object should look like and how it is tied to the underlying device memory.

We need some GPU memory allocated for registered a buffer with hipFile.

This isn't perfect, but it provides a starting point to at least running IO.

Not meant to be permanent. Just a quick and dirty script to verify running IO works.

Also adds limited typing hint for this method only. This import is only run if TYPE_CHECKING is defined. At runtime, if the ctypes package is not found, no error is raised since __future__.annotations is imported which treats the hint as a string literal and does not try to evaluate the name.

hipFileOpError and hipFileFileHandleType are exposed to the Python layer to provide the user direct access to these enum values.

A proper fix will come next where the C enum values are namespace qualified so that enum values exposed to Python do not need to have their names modified.

This puts the C hipFile symbols being imported by Cython into a "_c" namespace to avoid a namespace conflict when trying to set a Python variable/function with the same name.

The C functions hipFileRead()/hipFileWrite() report error conditions through a negative return value. If this return value is -1 or -5011, it indicates additional error information needs to be fetched as it cannot be included in a single return value. However, this error information needs to be fetched prior to control returning to Python. Otherwise there is no guarantee that these error values are still valid for the error that occurred.

Cleans up what is imported in __init__.py to not pollute the namespace when a user imports 'hipfile'.

Originally these functions were given different names when the initial Cython code was generated. In part, this was to avoid namespace conflicts. Now that the C functions are imported into their own "_c" namespace, we can reuse the same function names without conflict. Reusing the names improves consistency of what Cython function calls which C function.

Removes supporting Win32 from FileHandle. FileHandle was designed to manage opening & closing its own file reference. Python does not provide a public API to open a file which returns a Win32 handle. This leads to a contrived process of opening a file, getting a file descriptor, and converting it to a Win32 handle. To be clear, os.open() on Windows provides a POSIX FD. FileHandle does not currently support a user providing an already open file resource, which at this time seems to be the only plausible use case of a Win32 handle being used. For the sake of simplicity and getting out an initial release, we will just error if the OPAQUE_WIN32 handle type is set. (This however will leave Win32 support in the Cython layer as that is a trivial switch that gets passed into the C library.)

…evelop/ROCm_hipFile/pr-248

Copilot

Pull request overview

This PR adds an initial proof-of-concept Python package for hipFile, using Cython to bind a small subset of the hipFile C API and exposing minimal higher-level Python wrappers (Driver / FileHandle / Buffer).

Changes:

Introduces a hipfile Python package with Cython-backed low-level bindings plus thin Python wrapper classes/enums.
Adds scikit-build-core/CMake build scaffolding to compile and install the _hipfile extension module.
Adds an experimental README and a rough local test script.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
projects/hipfile/python/pyproject.toml	Defines the Python package metadata and scikit-build-core build backend.
projects/hipfile/python/CMakeLists.txt	CMake build logic to Cythonize and build the `_hipfile` extension and link to libhipfile.
projects/hipfile/python/README.md	Build/install instructions for the experimental bindings.
projects/hipfile/python/main.py	Manual smoke-test script exercising driver open, buffer registration, and read/write.
projects/hipfile/python/hipfile/init.py	Package exports and `__version__` derived from the C extension constants.
projects/hipfile/python/hipfile/_chipfile.pxd	Cython declarations for the subset of hipFile + HIP runtime APIs being wrapped.
projects/hipfile/python/hipfile/_hipfile.pyx	Cython wrapper layer that re-exports enums/constants and wraps key C APIs.
projects/hipfile/python/hipfile/enums.py	Python `IntEnum` mirrors for hipFile op errors and handle types.
projects/hipfile/python/hipfile/error.py	`HipFileException` and conversion to readable error strings.
projects/hipfile/python/hipfile/driver.py	Context-manager wrapper for hipFile driver open/close and use count.
projects/hipfile/python/hipfile/buffer.py	Context-manager wrapper for hipFile buffer register/deregister.
projects/hipfile/python/hipfile/file.py	Context-manager wrapper for file handle register/deregister plus sync read/write.
projects/hipfile/python/hipfile/properties.py	Small helpers for driver properties and version retrieval.
projects/hipfile/python/hipfile/hipMalloc.py	Temporary ctypes-based HIP malloc/free helper for early experimentation.
projects/hipfile/.gitignore	Ignores local Python venvs and `__pycache__` under the hipfile project.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

riley-dixon added 26 commits April 8, 2026 13:55

Python: Generate low-level Cython code

fcdb792

Ignore: Python venv's

0f7545e

Python/CMake: Define __HIP_PLATFORM_AMD__

f69ae0a

Python: Create Driver class

4e06092

Also move _hipfile driver functions out of public __init__.py

Python: Create a properties module.

2bb2d30

This module will contain the functions related to querying driver/library configuration/properties in a Python friendly manner.

Python: Create a FileHandle class.

013733a

A fairly rudimentary interface for opening/closing files with hipFile.

Python: Create a buffer module

ac4f6cd

A simple wrapper. Still need to figure out how exactly a hipFile Buffer object should look like and how it is tied to the underlying device memory.

Python: Add error handling to Driver.

fbaf27c

Python: Hack in hipMalloc & hipFree

66e9988

We need some GPU memory allocated for registered a buffer with hipFile.

Python: Add read/write methods to FileHandle

2cc15e5

This isn't perfect, but it provides a starting point to at least running IO.

Python: Add a hacked up testing script.

37fd240

Not meant to be permanent. Just a quick and dirty script to verify running IO works.

Python: Add a README for building the bindings

4984f6c

Python: Turn buffer functions into a class

bb7fa2a

Python: Change read()/write() to accept Buffer

6df21ad

Python: Expose 2 hipFile C enums to Python

02b3915

hipFileOpError and hipFileFileHandleType are exposed to the Python layer to provide the user direct access to these enum values.

Python: Temp fix C->Python enum namespace collisions

79589b0

A proper fix will come next where the C enum values are namespace qualified so that enum values exposed to Python do not need to have their names modified.

Python: FileHandle can now choose HandleType

43e86ff

Python: Fix importing C symbols in Cython

539c096

This puts the C hipFile symbols being imported by Cython into a "_c" namespace to avoid a namespace conflict when trying to set a Python variable/function with the same name.

Python: Cleanup __init__.py for public consumption

d8ecba1

Cleans up what is imported in __init__.py to not pollute the namespace when a user imports 'hipfile'.

Python: Formatting with Black

834c16f

Python: Formatting with pylint

57546b3

Python/CMake: Add single lint ignore for Python_add_library

737d1e6

systems-assistant bot added imported pr external contribution labels Apr 9, 2026

jordan-turbofish added the project: hipfile label Apr 9, 2026

jordan-turbofish changed the title ~~POC: hipFile Python API using Cython bindings~~ [hipFile] POC: hipFile Python API using Cython bindings Apr 9, 2026

riley-dixon added 2 commits April 9, 2026 17:03

Python: Address copilot review

3e7b8dd

riley-dixon force-pushed the import/develop/ROCm_hipFile/pr-248 branch from e366dc3 to f4a5cf9 Compare April 9, 2026 23:10

riley-dixon requested a review from a team as a code owner April 9, 2026 23:10

riley-dixon self-assigned this Apr 10, 2026

Merge commit '78aae81c4132234f0ce0395eaf7f16851411eca7' into import/d…

96726e1

…evelop/ROCm_hipFile/pr-248

riley-dixon force-pushed the import/develop/ROCm_hipFile/pr-248 branch from f4a5cf9 to 96726e1 Compare April 10, 2026 20:50

Merge branch 'develop' into import/develop/ROCm_hipFile/pr-248

d23eee8

gaoikawa requested a review from Copilot April 14, 2026 17:38

Copilot started reviewing on behalf of gaoikawa April 14, 2026 17:40 View session

Copilot AI reviewed Apr 14, 2026

View reviewed changes

riley-dixon changed the title ~~[hipFile] POC: hipFile Python API using Cython bindings~~ [hipFile] Python API using Cython bindings Apr 14, 2026

zbyrne approved these changes Apr 14, 2026

View reviewed changes

Resolve copilot suggestions

22bc5cc

riley-dixon force-pushed the import/develop/ROCm_hipFile/pr-248 branch from b5f49f5 to 22bc5cc Compare April 14, 2026 21:27

Merge branch 'develop' into import/develop/ROCm_hipFile/pr-248

e937057

riley-dixon merged commit 4dcdfd6 into develop Apr 14, 2026
9 checks passed

riley-dixon deleted the import/develop/ROCm_hipFile/pr-248 branch April 14, 2026 21:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[hipFile] Python API using Cython bindings#4865

[hipFile] Python API using Cython bindings#4865
riley-dixon merged 32 commits intodevelopfrom
import/develop/ROCm_hipFile/pr-248

systems-assistant bot commented Apr 9, 2026 •

edited by riley-dixon

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

systems-assistant bot commented Apr 9, 2026 • edited by riley-dixon Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Plan

Test Result

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

systems-assistant bot commented Apr 9, 2026 •

edited by riley-dixon

Loading