SIGABRT from `PQCRYPTO_RUST_randombytes`

[Rawa]

Hello!

In our app, MullvadVPN on android, We've seen rare crashes (SIGABRT) stemming from PQCRYPTO_RUST_randombytes on mostly Android TV devices. I believe one thing they have in common is that they run in a 32-bit mode, even though they may have 64 bit hardware. This has been true for each report I've seen so far.

Unfortunately, I have no way of reproducing this issue, we get about 100 crashes each month. So I don't have much to go on here. I've attached a bunch of reports from our Google Play Console below that might be of help.

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 6627 >>> net.mullvad.mullvadvpn <<<

backtrace:
  #00  pc 0x0000000000038f08  /apex/com.android.runtime/lib/bionic/libc.so (abort+172)
  #01  pc 0x00000000006ce00d  /data/app/~~LUfQapgBdk_T-IpOql5Ktw==/net.mullvad.mullvadvpn-nTFH_VjW7v-mvKczCOafiw==/lib/arm/libmullvad_jni.so (PQCRYPTO_RUST_randombytes) (BuildId: 8f7a222b11f6fd2dae186a7105ffd816)
  #02  pc 0x0000000000000004 

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 2504 >>> net.mullvad.mullvadvpn <<<

backtrace:
  #00  pc 0x00000000000388c0  /apex/com.android.runtime/lib/bionic/libc.so (abort+172)
  #01  pc 0x00000000006ce00d  /data/app/~~anxwKPWiuVxSBJTrndcxLg==/net.mullvad.mullvadvpn-jFyYe6wewF2EEECFP0hTrA==/lib/arm/libmullvad_jni.so (PQCRYPTO_RUST_randombytes) (BuildId: 8f7a222b11f6fd2dae186a7105ffd816)
  #02  pc 0x00000000000d4afe  [anon:scudo:secondary]

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 12078 >>> net.mullvad.mullvadvpn <<<

backtrace:
  #00  pc 0x0000000000062b20  /apex/com.android.runtime/lib/bionic/libc.so (abort+172)
  #01  pc 0x00000000006ce00d  /data/app/~~eSgcug1ApEDnwzunMFxgSA==/net.mullvad.mullvadvpn-aQa27glROI1Ha-nhM4lvVQ==/lib/arm/libmullvad_jni.so (PQCRYPTO_RUST_randombytes) (BuildId: 8f7a222b11f6fd2dae186a7105ffd816)
  #02  pc 0x0000000000000e3e  [anon:libc_malloc]

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 12172 >>> net.mullvad.mullvadvpn <<<

backtrace:
  #00  pc 0x000000000003bf0a  /apex/com.android.runtime/lib/bionic/libc.so (abort+138)
  #01  pc 0x00000000006ce00d  /data/app/~~vnEIJtB4yRyETmTrEgUuaQ==/net.mullvad.mullvadvpn-MVL15SBpDIIxDNN7skGyTQ==/lib/arm/libmullvad_jni.so (PQCRYPTO_RUST_randombytes) (BuildId: 8f7a222b11f6fd2dae186a7105ffd816)
  #02  pc 0x0000000000057042  [anon:scudo:primary]

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 6373 >>> net.mullvad.mullvadvpn <<<

backtrace:
  #00  pc 0x0000000000038f08  /apex/com.android.runtime/lib/bionic/libc.so (abort+172)
  #01  pc 0x00000000006ce00d  /data/app/~~LUfQapgBdk_T-IpOql5Ktw==/net.mullvad.mullvadvpn-nTFH_VjW7v-mvKczCOafiw==/lib/arm/libmullvad_jni.so (PQCRYPTO_RUST_randombytes) (BuildId: 8f7a222b11f6fd2dae186a7105ffd816)
  #02  pc 0x0000000000030d53  /apex/com.android.runtime/lib/bionic/libc.so (scudo::Allocator<scudo::AndroidConfig, &(scudo_malloc_postinit)>::quarantineOrDeallocateChunk(scudo::Options, void*, scudo::Chunk::UnpackedHeader*, unsigned long)+238)
  #03  pc 0x0000000000000005

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 7098 >>> net.mullvad.mullvadvpn <<<

backtrace:
  #00  pc 0x000000000003b5f6  /apex/com.android.runtime/lib/bionic/libc.so (abort+138)
  #01  pc 0x000000000070b741  /data/app/~~pQOaAJE_S_waD1AQtEzKnA==/net.mullvad.mullvadvpn-CtKDJ9ipENHDWfUpecHNng==/lib/arm/libmullvad_jni.so (PQCRYPTO_RUST_randombytes) (BuildId: 1ddd96751f993fea3ff65308193ed8c7)
  #02  pc 0x000000000001b17a  [anon:dalvik-concurrent copying gc mark stack]

Let me know if there is anything I can assist with helping debugging this issue.

[thomwiggers]

I don't think that this is enough information to work out what is going on. The function itself is as simple as:

pqcrypto/pqcrypto-internals/src/lib.rs

Lines 22 to 27 in 4d95901

	#[no_mangle]
	pub unsafe extern "C" fn PQCRYPTO_RUST_randombytes(buf: *mut u8, len: size_t) -> c_int {
	let buf = slice::from_raw_parts_mut(buf, len);
	getrandom::fill(buf).expect("RNG Failed");
	0
	}

So then the question is if any of its callers (i.e., the primitives) are accessing it in some way that is illegal. But the implementations in PQClean should do valgrind testing, including on 32-bit targets...

[Rawa]

I don't think that this is enough information to work out what is going on. The function itself is as simple as:

pqcrypto/pqcrypto-internals/src/lib.rs

Lines 22 to 27 in 4d95901

	#[no_mangle]
	pub unsafe extern "C" fn PQCRYPTO_RUST_randombytes(buf: *mut u8, len: size_t) -> c_int {
	let buf = slice::from_raw_parts_mut(buf, len);
	getrandom::fill(buf).expect("RNG Failed");
	0
	}

So then the question is if any of its callers (i.e., the primitives) are accessing it in some way that is illegal. But the implementations in PQClean should do valgrind testing, including on 32-bit targets...

Thanks for the quick response, we have the same hardware that crashes, units that run in 32bit mode, but we've yet to be able to reproduce it manually. I'll try and put some more effort into reproducing next week. To see if I can get additional information.

[Rawa]

I tried reproducing it, but have not been successful. I generated more than 40000 keypairs on the hardware w/o any problem. We don't really know if it is from when generating the keypairs the problem arises either, however, that code allocates a lot of stack and then randombytes are the first function to be invoked afterwards that actually touches it.

So one theory right now is that we possibly are out of stack on these devices, and it crashes on PQCRYPTO_RUST_randombytes because that is the first time the memory is accessed. But it feels like that should result in a SIGSEGV instead.

Another theory would be as you mentioned that some of the input to the PQCRYPTO_RUST_randombytes function is illegal when used by PQCLEAN.

A third option is that getrandom actually fails on these devices sometimes, which causes a panic and could result in a SIGABRT. According to docs, one can be "reasonably confident" that an error will not occur if the method called worked previously. So I guess that it could be that on these 32bit Android devices they sometimes happen to fail.

[Rawa]

Update regarding the third alternative, we've verified that a panic, e.g the .expect("RNG Failed"); would cause the app to crash with a SIGABRT, similar to what we see in our reports from the play console.

Without reproducing the issue it is hard to determine exactly what the root cause of the problem is. It could be any of the issues above, or something completely different. We are going to spend some more time trying to reproduce the issue, if you have any ideas of what we could try let us know. If you don't think the issue lies within this library feel free to close this issue. It is hard to move on w/o more information to go on.

[thomwiggers]

It's plausible that the getrandom is failing if the device has just booted, with small microcontroller boards like Raspberry Pi's I've seen those sorts of issues before. I don't think that my library has a good way of recovering from this though. A workaround may be to ensure that getrandom() returns success before calling any rustpq functions (once it does, it should keep doing so on modern linux at least).

[Rawa]

be to ensure that getrandom() returns success before calling any rustpq functions (once it does, it should keep doing so on modern linux at least).

Oh, that is interesting. We do offer a "connect on start-up" feature on TV devices. However, some mobile phones are affected as well, but they too have the option to have a "Always-on" VPN which effectively would do the same.

A colleague also told me that Rust announched 1.92.0 that will "Emit unwind tables even when -Cpanic=abort is enabled on linux", maybe this might give us better feedback from the crashes as well.