Dual 4090 report ‘Cuda failure p2pBandwidthLatencyTest.cu:189: 'mapping of buffer object failed'’ in Third Gen Xeon platform with PCIe Gen4 switch

[aosudh]

NVIDIA Open GPU Kernel Modules Version

550.90.07-p2p, 565.57.01-p2p

Please confirm this issue does not happen with the proprietary driver (of the same version). This issue tracker is only for bugs specific to the open kernel driver.

• I confirm that this does not happen with the proprietary driver package.

Operating System and Version

Ubuntu 22.04.2 LTS, Ubuntu 24.04.2 LTS

Kernel Release

Linux 4090 6.8.0-55-generic NVIDIA#57-Ubuntu SMP PREEMPT_DYNAMIC Wed Feb 12 23:42:21 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Please confirm you are running a stable release kernel (e.g. not a -rc). We do not accept bug reports for unreleased kernels.

• I am running on a stable kernel release.

Hardware: GPU

RTX4090

Describe the bug

After both the system driver and the kernel driver are installed, run nvidia-smi topo -p2p p. The driver shows that the device supports P2P.

The results obtained using non - P2P drivers are also shown in the following figure.

However, whether in Docker or on the host machine, when using p2pbandwidthtest and simpleP2P test compiled with CUDA 12.4 and 12.6 and their corresponding CUDA samples, the P2P status can be recognized correctly, but the P2P test cannot be carried out.

To Reproduce

Install the driver with the --no-kernel-module option. Run install.sh. Install CUDA from the runfile. Install CUDA samples, compile them, and then run.

Bug Incidence

Always

nvidia-bug-report.log.gz

nvidia-bug-report.log.gz

More Info

No response

[lambo111-x86]

+1

[zvorinji]

This is the expected behavior behind a switch. The Linux kernel does not support it yet. There is some work on this topic going on but I think it's probably 6-12 months before it's done if ever. And even so, it's unclear if the p2p driver will work out of the box or ever from those changes.

[hyson666]

Same problem, has anyone solved this problem?

[developer-hq]

same problem

[caolonghao]

same problem

[cunfate]

I've got the same problem and after that I opened the resizeable bar in BIOS, and this problem sloved.
If you want to check this, run lspci -s <gpu-bdf> -v, the bar 1 size should be 32GB, else it will be 256MB with resizeable bar disabled.

[changzhi1990]

I've got the same problem and after that I opened the resizeable bar in BIOS, and this problem sloved. If you want to check this, run lspci -s <gpu-bdf> -v, the bar 1 size should be 32GB, else it will be 256MB with resizeable bar disabled.

Hi, Does the Region 0 or 1 should be 32GB? Here is the device info below:

root@test:/home/test/cuda-samples-12.8/build/Samples/5_Domain_Specific# lspci -vv -s E1:00.0
e1:00.0 VGA compatible controller: NVIDIA Corporation Device 2b85 (rev a1) (prog-if 00 [VGA controller])
Subsystem: CardExpert Technology Device f319
Physical Slot: 35
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 362
NUMA node: 0
IOMMU group: 55
Region 0: Memory at b8000000 (32-bit, non-prefetchable) [size=64M]
Region 1: Memory at 644f0000000 (64-bit, prefetchable) [size=256M]
Region 3: Memory at 64510000000 (64-bit, prefetchable) [size=32M]
Region 5: I/O ports at f000 [size=128]

[cunfate]

I've got the same problem and after that I opened the resizeable bar in BIOS, and this problem sloved. If you want to check this, run lspci -s <gpu-bdf> -v, the bar 1 size should be 32GB, else it will be 256MB with resizeable bar disabled.

Hi, Does the Region 0 or 1 should be 32GB? Here is the device info below:

root@test:/home/test/cuda-samples-12.8/build/Samples/5_Domain_Specific# lspci -vv -s E1:00.0 e1:00.0 VGA compatible controller: NVIDIA Corporation Device 2b85 (rev a1) (prog-if 00 [VGA controller]) Subsystem: CardExpert Technology Device f319 Physical Slot: 35 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx- Latency: 0 Interrupt: pin A routed to IRQ 362 NUMA node: 0 IOMMU group: 55 Region 0: Memory at b8000000 (32-bit, non-prefetchable) [size=64M] Region 1: Memory at 644f0000000 (64-bit, prefetchable) [size=256M] Region 3: Memory at 64510000000 (64-bit, prefetchable) [size=32M] Region 5: I/O ports at f000 [size=128]

Region 1 should be 32GB.

[cunfate]

I've got the same problem and after that I opened the resizeable bar in BIOS, and this problem sloved. If you want to check this, run lspci -s <gpu-bdf> -v, the bar 1 size should be 32GB, else it will be 256MB with resizeable bar disabled.

Hi, Does the Region 0 or 1 should be 32GB? Here is the device info below:

root@test:/home/test/cuda-samples-12.8/build/Samples/5_Domain_Specific# lspci -vv -s E1:00.0 e1:00.0 VGA compatible controller: NVIDIA Corporation Device 2b85 (rev a1) (prog-if 00 [VGA controller]) Subsystem: CardExpert Technology Device f319 Physical Slot: 35 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- SERR- <PERR- INTx- Latency: 0 Interrupt: pin A routed to IRQ 362 NUMA node: 0 IOMMU group: 55 Region 0: Memory at b8000000 (32-bit, non-prefetchable) [size=64M] Region 1: Memory at 644f0000000 (64-bit, prefetchable) [size=256M] Region 3: Memory at 64510000000 (64-bit, prefetchable) [size=32M] Region 5: I/O ports at f000 [size=128]

Resizeable bar seems not enabled on your platform because the size of Region 1 is 256MB. Consider update you BIOS config to enable Resizeable Bar or update your BIOS or firmware version to support it. It may be called "Resize-Bar" or "Resizeble Bar" in BIOS configuration.

[zhangboyue]

This is the expected behavior behind a switch. The Linux kernel does not support it yet. There is some work on this topic going on but I think it's probably 6-12 months before it's done if ever. And even so, it's unclear if the p2p driver will work out of the box or ever from those changes.

Hi @zvorinji , pretty appreciate for the info. I'm wondering if there's any info about the discussion over the switch topic ? Currently, I'm blocked at this point in a system with a switch, and would like to follow up the topic.

[BluffeyTest]

I've got the same problem and after that I opened the resizeable bar in BIOS, and this problem sloved. If you want to check this, run lspci -s <gpu-bdf> -v, the bar 1 size should be 32GB, else it will be 256MB with resizeable bar disabled.

it shows 32G, but still not work.

VGA compatible controller: NVIDIA Corporation Device 2684 (rev a1) (prog-if 00 [VGA controller])
        Subsystem: NVIDIA Corporation Device 16f3
        Flags: bus master, fast devsel, latency 0, IRQ 129, NUMA node 1, IOMMU group 7
        Memory at df000000 (32-bit, non-prefetchable) [size=16M]
        Memory at 397000000000 (64-bit, prefetchable) [size=32G]
        Memory at 397800000000 (64-bit, prefetchable) [size=32M]
        I/O ports at d000 [size=128]
        Expansion ROM at e0000000 [virtual] [disabled] [size=512K]
        Capabilities: <access denied>
        Kernel driver in use: nvidia
        Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia