GPU Support

[ananay-22]

I'm wondering if there ever is a plan to support Native backed GPU piplines (thinking more EGL/ Vulkan).

I understand if not since that will blow things up complexity wise but if it is on the roadmap I would love to help since I have to work on EGL + AVFoudation (on apple) GPU camera pipelines at some point on a project that would depend on this for CPU pipelines

[wysaid]

Sure! I have considered adding a similar implementation, but we need to carefully think about the complexity of the interface. For example, where should this layer be implemented? Taking AVFoundation as you mentioned, it supports direct output of RGB/RGBA as well as NV12 format images, but the efficiency of these two formats is quite different. Should we support both? How should the interface be designed (for example, NV12 may require outputting two GPU textures), etc.

If you have a good design or idea, feel free to submit a PR or share your thoughts! We can work on this together.

[ananay-22]

I'm still playing around with the CPU API - will familiarize with it then try and come up with a design decision.
The way I view it - ccap should abstract the native camera initialization, not the gpu copies etc. The camera provider should stay the same and as long as you specify a gpu driver (directx, avfoundation, egl, vulkan etc) you should have to as a user specify (not sure if this should be a callback or a delegate or something) how they would like to pull the data out (which is especially important since if you are using GPUs for this you're probably trying to minimize copies).

So the way I see it - you're only killing boilerplate on one side (how to setup the camera) and not the other (how to pull it into your GPU context).

From what I can tell - the CPU API does abstract some of this (it maintains it's own pool? I see an allocator on the VideoFrame that's why im assuming this)

[wysaid]

That makes total sense. Your design aligns with what VideoFrame already provides — nativeHandle carries the platform-native buffer (CVPixelBufferRef on Apple, IMFSample on MSMF, etc.). If you set PixelFormatInternal to a format the camera natively supports and also set PixelFormatOutput to the same format, the conversion step is skipped entirely and data points directly into the capture buffer. From there you can pull the raw buffer in your callback and handle GPU upload however you like.

The allocator you saw on VideoFrame is just for CPU fallback when zero-copy isn't possible — you can ignore it in a GPU path.

Some things I'm thinking about going forward:

• Short term: Add an example demonstrating the callback + raw buffer pattern for GPU use cases
• Long term: Expose richer native handles per backend (e.g. DMA-BUF fd on Linux, D3D11 texture via IMFDXGIBuffer on Windows MSMF) so users can do true zero-copy GPU import without touching platform APIs directly

If you have more ideas, let's discuss together. If you already have a good implementation, feel free to submit a PR directly 😃