Fix BPF perf event array creation failure on glibc 2.36+

[malladisiddu]

Summary

Replace get_nprocs_conf() with get_nprocs() in perfeventarray.cpp to fix BPF initialization failures on systems running glibc 2.36+ where the reported "configured" CPU count exceeds the actual online CPU count.

---

Problem Description

BPF-based applications (i.e., in my case it is osquery) using ebpf-common fail to initialize on Ubuntu 24.04 (glibc 2.39) with the following error:

Failed to create the perf event array: Failed to create the perf BPF output

Root Cause

The PerfEventArray::create() function uses get_nprocs_conf() to determine the number of CPUs for perf event array creation.

glibc 2.36 changed the behavior of get_nprocs_conf():

glibc Version	get_nprocs_conf() reads from	Behavior
≤ 2.35	CPU directories (/sys/devices/system/cpu/cpu*)	Returns actual CPU count
≥ 2.36	/sys/devices/system/cpu/possible	Returns maximum possible CPUs

Both approaches are valid interpretations of "configured processors." The issue is that the code using this function assumed the returned value would always represent CPUs usable with perf_event_open().

Per the man page, get_nprocs_conf() returns "the number of processors configured by the operating system" while get_nprocs() returns "the number of processors currently available in the system." On systems where configured CPUs exceed online CPUs, using get_nprocs_conf() causes failures when attempting to create perf events for CPUs that are not currently available.

---

strace Output (Failing System)

perf_event_open({type=PERF_TYPE_SOFTWARE, ...}, -1, 0, -1, 0) = 16   # CPU 0: Success
perf_event_open({type=PERF_TYPE_SOFTWARE, ...}, -1, 1, -1, 0) = 17   # CPU 1: Success
perf_event_open({type=PERF_TYPE_SOFTWARE, ...}, -1, 2, -1, 0) = 18   # CPU 2: Success
..
..
perf_event_open({type=PERF_TYPE_SOFTWARE, ...}, -1, 7, -1, 0) = 23   # CPU 7: Success
perf_event_open({type=PERF_TYPE_SOFTWARE, ...}, -1, 8, -1, 0) = -1 ENODEV (No such device)  # CPU 8: FAILS

CPU Configuration (Affected System)

$ getconf _NPROCESSORS_CONF
240                                    #  Incorrect (from /sys/devices/system/cpu/possible)

$ getconf _NPROCESSORS_ONLN
8                                      #  Correct (actual online CPUs)

$ cat /sys/devices/system/cpu/online
0-7

$ cat /sys/devices/system/cpu/possible
0-239

---

Testing

Before Fix

Event publisher not enabled: BPFEventPublisher: Failed to create the perf event array

After Fix

Initialized BPF probe for syscall fork
Initialized BPF probe for syscall vfork
Initialized BPF probe for syscall clone
Initialized BPF probe for syscall execve
...
Starting event publisher run loop: BPFEventPublisher

Impact

Affected Systems

• Any system where:
  • glibc version is 2.36 or higher (Ubuntu 22.10+, Ubuntu 24.04+ etc.)
  • Configured CPUs (get_nprocs_conf()) exceeds available CPUs (get_nprocs())

Downstream Projects Affected

• osquery (via ebpfpub)
• Any other projects using ebpf-common for BPF functionality

---

References

• glibc commit introducing the change (2022)
• glibc mailing list discussion
• glibc get_nprocs_conf implementation
• perf_event_open man page

---

Additional Notes

This issue was discovered while debugging osquery's BPFEventPublisher failures on Ubuntu 24.04. The fix has been validated in production environments.
If maintainers prefer an alternative approach, I'm happy to discuss and implement accordingly.

[CLAassistant]

All committers have signed the CLA.

[alessandrogario]

Summary

Replace get_nprocs_conf() with get_nprocs() in perfeventarray.cpp to fix BPF initialization failures on systems running glibc 2.36+ where the reported "configured" CPU count exceeds the actual online CPU count.

Problem Description

BPF-based applications (i.e., in my case it is osquery) using ebpf-common fail to initialize on Ubuntu 24.04 (glibc 2.39) with the following error:

Failed to create the perf event array: Failed to create the perf BPF output

Root Cause

The PerfEventArray::create() function uses get_nprocs_conf() to determine the number of CPUs for perf event array creation.

glibc 2.36 changed the behavior of get_nprocs_conf():

glibc Version get_nprocs_conf() reads from Behavior
≤ 2.35 CPU directories (/sys/devices/system/cpu/cpu*) Returns actual CPU count
≥ 2.36 /sys/devices/system/cpu/possible Returns maximum possible CPUs
Both approaches are valid interpretations of "configured processors." The issue is that the code using this function assumed the returned value would always represent CPUs usable with perf_event_open().

Per the man page, get_nprocs_conf() returns "the number of processors configured by the operating system" while get_nprocs() returns "the number of processors currently available in the system." On systems where configured CPUs exceed online CPUs, using get_nprocs_conf() causes failures when attempting to create perf events for CPUs that are not currently available.

strace Output (Failing System)

perf_event_open({type=PERF_TYPE_SOFTWARE, ...}, -1, 0, -1, 0) = 16   # CPU 0: Success
perf_event_open({type=PERF_TYPE_SOFTWARE, ...}, -1, 1, -1, 0) = 17   # CPU 1: Success
perf_event_open({type=PERF_TYPE_SOFTWARE, ...}, -1, 2, -1, 0) = 18   # CPU 2: Success
..
..
perf_event_open({type=PERF_TYPE_SOFTWARE, ...}, -1, 7, -1, 0) = 23   # CPU 7: Success
perf_event_open({type=PERF_TYPE_SOFTWARE, ...}, -1, 8, -1, 0) = -1 ENODEV (No such device)  # CPU 8: FAILS

### CPU Configuration (Affected System) ```shell $ getconf _NPROCESSORS_CONF 240 # Incorrect (from /sys/devices/system/cpu/possible)

$ getconf _NPROCESSORS_ONLN
8 # Correct (actual online CPUs)

$ cat /sys/devices/system/cpu/online
0-7

$ cat /sys/devices/system/cpu/possible
0-239

<img alt="cpu_configuration_mismatch" width="1692" height="408" src="https://private-user-images.githubusercontent.com/46401317/546130655-cca394af-7db3-44f6-b76b-10664b42fed4.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NzQxOTM2ODUsIm5iZiI6MTc3NDE5MzM4NSwicGF0aCI6Ii80NjQwMTMxNy81NDYxMzA2NTUtY2NhMzk0YWYtN2RiMy00NGY2LWI3NmItMTA2NjRiNDJmZWQ0LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNjAzMjIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjYwMzIyVDE1Mjk0NVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTRiODg2ZmQ2MGEyZjliNjA1MzY5N2M1OWM0NzZmZmVhZTc0NWYwYTljNzFmYTIwODJmYWRjNGE4MTQ5ZDc5NjImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.RMSqWRLl34zDbyVO0us7O7QJfslCMapAW510DG2ZotM">
## Testing
### Before Fix

Event publisher not enabled: BPFEventPublisher: Failed to create the perf event array

<img alt="bpf_failure_log" width="2000" height="758" src="https://private-user-images.githubusercontent.com/46401317/546130843-cdcd2bfd-8aaf-4ae5-9204-2285262c38c2.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NzQxOTM2ODUsIm5iZiI6MTc3NDE5MzM4NSwicGF0aCI6Ii80NjQwMTMxNy81NDYxMzA4NDMtY2RjZDJiZmQtOGFhZi00YWU1LTkyMDQtMjI4NTI2MmMzOGMyLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNjAzMjIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjYwMzIyVDE1Mjk0NVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWY0Mjk2YTVkYjM0NTlkOTcyMjk2NzJjYjY3ZjY5YTBjYjg0ZjZiOTUxYWRlMTc1OWVkNGExZTRlZTdiZWQ3NTUmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.OOKi96-YJPs3y1M4jcPgFHAQKKPrOHllWUthUQjHXl8">
### After Fix

Initialized BPF probe for syscall fork
Initialized BPF probe for syscall vfork
Initialized BPF probe for syscall clone
Initialized BPF probe for syscall execve
...
Starting event publisher run loop: BPFEventPublisher

<img alt="bpf_success_log" width="2000" height="1486" src="https://private-user-images.githubusercontent.com/46401317/546130961-34cb6e5c-f7c2-4505-90c6-80de4ca74bf9.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NzQxOTM2ODUsIm5iZiI6MTc3NDE5MzM4NSwicGF0aCI6Ii80NjQwMTMxNy81NDYxMzA5NjEtMzRjYjZlNWMtZjdjMi00NTA1LTkwYzYtODBkZTRjYTc0YmY5LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNjAzMjIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjYwMzIyVDE1Mjk0NVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTI3ZDAwNjQ1OWNjNGRjODdiY2Y2ODI0NjVjMDVjYzM5YTViNWJiMGMwZWE4NTkyNDkxMzZlMmI5ZDk1ODc5YTEmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.xfNcbwhwh7uvwRHatbA80yjKI0hJnCP-LosJFBU5i4w">
## Impact
### Affected Systems
* Any system where:
  
  * glibc version is 2.36 or higher (Ubuntu 22.10+, Ubuntu 24.04+ etc.)
  * Configured CPUs (`get_nprocs_conf()`) exceeds available CPUs (`get_nprocs()`)

### Downstream Projects Affected
* [osquery](https://github.com/osquery/osquery) (via ebpfpub)
* Any other projects using ebpf-common for BPF functionality

## References
* [glibc commit introducing the change (2022)](https://sourceware.org/pipermail/glibc-cvs/2022q2/078952.html)
* [glibc mailing list discussion](https://sourceware.org/pipermail/libc-alpha/2022-August/141193.html)
* [glibc get_nprocs_conf implementation](https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/unix/sysv/linux/getsysstats.c)
* [perf_event_open man page](https://man7.org/linux/man-pages/man2/perf_event_open.2.html)

## Additional Notes
This issue was discovered while debugging osquery's BPFEventPublisher failures on Ubuntu 24.04. The fix has been validated in production environments. If maintainers prefer an alternative approach, I'm happy to discuss and implement accordingly.

Hello @malladisiddu, thanks for looking into this issue! Could you take a look at PR #41 instead? It implements better CPU detection, since the CPU topology on the machine allows for holes in the CPU index list

[malladisiddu]

Hey @alessandrogario, your #41 is a much better approach, reading /sys/devices/system/cpu/online directly handles the non-contiguous CPU case that my simple get_nprocs() swap doesn't cover. Happy to close this one.

Thanks for the quick and thorough fix!