Skip to content

Conversation

@illwieckz
Copy link
Member

@illwieckz illwieckz commented Dec 9, 2025

Extracted from:

Disclosure: this 6-characters fix is the result of me toying with ChatGPT as I used that bug for an experiment:

--- a/src/trusted/service_runtime/linux/nacl_bootstrap.x
+++ b/src/trusted/service_runtime/linux/nacl_bootstrap.x
@@ -78,7 +78,7 @@ SECTIONS {
   .rodata : {
     *(.rodata*)
     *(.eh_frame*)
-  }
+  } :text
 
   etext = .;
 

The bug as I experienced it 8 months ago was that the nacl_helper_bootstrap was segfaulting when built with GCC, not when built with Clang. At the time I identified the bug I could build it with GCC up to GCC 12 in an old Debian chroot. On a more recent Ubuntu today, I could not build the bootstrap helper with any GCC version, including some as old as GCC 9.

So out of curiosity I did some experiments with ChatGPT, submitting the results of my existing discoveries done in my previous debugging sessions, the description of how I built the tool and some dump of built executable metadata, and even an hexdump of the bogus binary itself. I kept secret the fact the old GCCs that were building properly the binary were running on an older system with older binaries in PATH.

ChatGPT suggested this fix, explaining that supposedly a ld.bfd change happened in binutils modifying the behavior, the explanation proposed was that the .rodata section isn't assigned to any PHDR because the .rodata layout has no PHDR tag, so with older binutils the .rodata was implicitly attached to the previous PHDR, but that with newer binutils it wasn't, so the _start entry point would jump to unmapped memory and crash. This explanation is consistent with my observation where I noticed in debugger that it was crashing early on _start. The suggestion of a behavioral change due to a different environment (which I had not disclosed) and not in GCC is also consistent with my observation.

It is suggested that the fact the code runs when built with Clang is fortuitous.

I verified that with this small change the bootstrap helper not only builds with GCC but also runs properly.

@slipher
Copy link
Member

slipher commented Dec 10, 2025

That does seem like an ld bug based on the manual.

If you place a section in one or more segments using ‘:phdr’, then the linker will place all subsequent allocatable sections which do not specify ‘:phdr’ in the same segments.

-- https://sourceware.org/binutils/docs/ld/PHDRS.html

@slipher
Copy link
Member

slipher commented Dec 10, 2025

Building with GCC and Scons seems to work fine for me. Tests pass and the loaders work with Daemon. For example scons --mode=opt-host platform=x86-64 sel_ldr werror=0 --no-clang --verbose (note the --no-clang). So maybe it's just an issue with the CMake code.

@illwieckz
Copy link
Member Author

illwieckz commented Dec 10, 2025

Add this file as build.sh in src/trusted/service_runtime/linux:

#! /usr/bin/env bash

set -u -e

export LANGUAGE='C.UTF-8'
export LANG="${LANGUAGE}"

CC="${CC:-cc}"
PYTHON="${PYTHON:-python3}"

declare -a CCARGS
CCARGS+=(-static)
CCARGS+=(-D_FORTIFY_SOURCE=2)
CCARGS+=(-D_GNU_SOURCE=1)
CCARGS+=(-D_LARGEFILE64_SOURCE=1)
CCARGS+=(-D_POSIX_C_SOURCE=199506)
CCARGS+=(-D_XOPEN_SOURCE=600)
CCARGS+=(-U__STDC_HOSTED__)
CCARGS+=(-D__STDC_HOSTED__=1)
CCARGS+=(-D__STDC_LIMIT_MACROS=1)
CCARGS+=(-D__STDC_FORMAT_MACROS=1)
CCARGS+=(-I ../../../../include-hax)
CCARGS+=(-std=gnu99)
CCARGS+=(-fno-pic -fno-PIC)
CCARGS+=(-fno-pie -fno-PIE)
CCARGS+=(-fvisibility=hidden)
CCARGS+=(-fno-stack-protector)
CCARGS+=(-ffreestanding)
CCARGS+=(-Wno-builtin-macro-redefined)
CCARGS+=(--param ssp-buffer-size=4)
CCARGS+=(-Os)
CCARGS+=(-c)
CCARGS+=(-o nacl_bootstrap.c.o)
CCARGS+=(nacl_bootstrap.c)

declare -a BFDARGS
BFDARGS+=(--compiler "${CC}")
BFDARGS+=(-m 'elf_x86_64')
BFDARGS+=(--build-id)
BFDARGS+=(--static)
BFDARGS+=(-z 'max-page-size=0x1000')
BFDARGS+=(--defsym RESERVE_TOP='0x0')
BFDARGS+=(--script nacl_bootstrap.x)
BFDARGS+=(-o nacl_bootstrap_raw)
BFDARGS+=(nacl_bootstrap.c.o)

rm -f nacl_bootstrap.c.o nacl_bootstrap_raw nacl_helper_bootstrap

set -x

"${CC}" "${CCARGS[@]}" ${CFLAGS:-}

"${PYTHON}" ./ld_bfd.py "${BFDARGS[@]}"

"${PYTHON}" ./nacl_bootstrap_munge_phdr.py nacl_bootstrap_raw nacl_helper_bootstrap

Then run it this way:

CC=gcc ./build.sh && ./nacl_helper_bootstrap

If it still works, then you have a lucky environment.

What scons does is hard to trace.

The script is made to facilitate the removal or the adding of flags for testing them.

@slipher
Copy link
Member

slipher commented Dec 10, 2025

What's the point of using this ld_bfd.py? I'm pretty sure that is not involved in the actual build since my system does not have /usr/bin/python...

@illwieckz
Copy link
Member Author

This is extracted from what scons does.

@slipher
Copy link
Member

slipher commented Dec 10, 2025

I see, thanks. I confirm that it depends on the toolchain version and the result is the same with scons or your script. Broken with Arch Linux (binutils 2.45.1) but worked with 2.44

@illwieckz
Copy link
Member Author

illwieckz commented Dec 11, 2025

In fact it doesn't look to be that related to binutils:

compiler binutils distro master patch
GCC 5 2.30 Ubuntu 18.04 Bionic ✅️ working ✅️ working
GCC 6 2.30 Ubuntu 18.04 Bionic ✅️ working ✅️ working
GCC 7 2.30 Ubuntu 18.04 Bionic ✅️ working ✅️ working
GCC 8 2.30 Ubuntu 18.04 Bionic ✅️ working ✅️ working
Clang 8 2.30 Ubuntu 18.04 Bionic ✅️ working ✅️ working
Clang 9 2.30 Ubuntu 18.04 Bionic ✅️ working ✅️ working
Clang 10 2.30 Ubuntu 18.04 Bionic ✅️ working ✅️ working
GCC 7 2.31.1 Debian 10 Buster ✅️ working ✅️ working
GCC 8 2.31.1 Debian 10 Buster ✅️ working ✅️ working
Clang 7 2.31.1 Debian 10 Buster ✅️ working ✅️ working
Clang 8 2.31.1 Debian 10 Buster ✅️ working ✅️ working
Clang 11 2.31.1 Debian 10 Buster ✅️ working ✅️ working
Clang 13 2.31.1 Debian 10 Buster ✅️ working ✅️ working
GCC 7 2.34 Ubuntu 20.04 Focal ✅️ working ✅️ working
GCC 8 2.34 Ubuntu 20.04 Focal ❌️ broken ✅️ working
GCC 9 2.34 Ubuntu 20.04 Focal ❌️ broken ✅️ working
GCC 10 2.34 Ubuntu 20.04 Focal ❌️ broken ✅️ working
Clang 7 2.34 Ubuntu 20.04 Focal ✅️ working ✅️ working
Clang 8 2.34 Ubuntu 20.04 Focal ✅️ working ✅️ working
Clang 9 2.34 Ubuntu 20.04 Focal ✅️ working ✅️ working
Clang 10 2.34 Ubuntu 20.04 Focal ✅️ working ✅️ working
Clang 11 2.34 Ubuntu 20.04 Focal ✅️ working ✅️ working
Clang 12 2.34 Ubuntu 20.04 Focal ✅️ working ✅️ working
Clang 18 2.34 Ubuntu 20.04 Focal ✅️ working ✅️ working
GCC 9 2.35.2 Debian 11 Bullseye ✅️ working ✅️ working
GCC 10 2.35.2 Debian 11 Bullseye ✅️ working ✅️ working
Clang 11 2.35.2 Debian 11 Bullseye ✅️ working ✅️ working
Clang 13 2.35.2 Debian 11 Bullseye ✅️ working ✅️ working
GCC 10 2.38 Ubuntu 22.04 Jammy ❌️ broken ✅️ working
GCC 11 2.38 Ubuntu 22.04 Jammy ❌️ broken ✅️ working
GCC 12 2.38 Ubuntu 22.04 Jammy ❌️ broken ✅️ working
Clang 11 2.38 Ubuntu 22.04 Jammy ✅️ working ✅️ working
Clang 12 2.38 Ubuntu 22.04 Jammy ✅️ working ✅️ working
Clang 13 2.38 Ubuntu 22.04 Jammy ✅️ working ✅️ working
Clang 14 2.38 Ubuntu 22.04 Jammy ✅️ working ✅️ working
GCC 10 2.40 Debian 12 Bookworm ✅️ working ✅️ working
GCC 12 2.40 Debian 12 Bookworm ✅️ working ✅️ working
Clang 11 2.40 Debian 12 Bookworm ✅️ working ✅️ working
Clang 13 2.40 Debian 12 Bookworm ✅️ working ✅️ working
Clang 14 2.40 Debian 12 Bookworm ✅️ working ✅️ working
Clang 15 2.40 Debian 12 Bookworm ✅️ working ✅️ working
Clang 16 2.40 Debian 12 Bookworm ✅️ working ✅️ working
Clang 19 2.40 Debian 12 Bookworm ✅️ working ✅️ working
GCC 10 2.42 Ubuntu 24.04 Noble ❌️ broken ✅️ working
GCC 11 2.42 Ubuntu 24.04 Noble ❌️ broken ✅️ working
GCC 12 2.42 Ubuntu 24.04 Noble ❌️ broken ✅️ working
GCC 13 2.42 Ubuntu 24.04 Noble ❌️ broken ✅️ working
GCC 14 2.42 Ubuntu 24.04 Noble ❌️ broken ✅️ working
Clang 10 2.42 Ubuntu 24.04 Noble ✅️ working ✅️ working
Clang 11 2.42 Ubuntu 24.04 Noble ✅️ working ✅️ working
Clang 12 2.42 Ubuntu 24.04 Noble ✅️ working ✅️ working
Clang 13 2.42 Ubuntu 24.04 Noble ✅️ working ✅️ working
Clang 14 2.42 Ubuntu 24.04 Noble ✅️ working ✅️ working
Clang 15 2.42 Ubuntu 24.04 Noble ✅️ working ✅️ working
Clang 16 2.42 Ubuntu 24.04 Noble ✅️ working ✅️ working
Clang 17 2.42 Ubuntu 24.04 Noble ✅️ working ✅️ working
Clang 18 2.42 Ubuntu 24.04 Noble ✅️ working ✅️ working
Clang 19 2.42 Ubuntu 24.04 Noble ✅️ working ✅️ working
Clang 20 2.42 Ubuntu 24.04 Noble ✅️ working ✅️ working
GCC 10 2.44 Debian 13 Trixie ✅️ working ✅️ working
GCC 12 2.44 Debian 13 Trixie ✅️ working ✅️ working
GCC 14 2.44 Debian 13 Trixie ✅️ working ✅️ working
Clang 11 2.44 Debian 13 Trixie ✅️ working ✅️ working
Clang 13 2.44 Debian 13 Trixie ✅️ working ✅️ working
Clang 14 2.44 Debian 13 Trixie ✅️ working ✅️ working
Clang 17 2.44 Debian 13 Trixie ✅️ working ✅️ working
Clang 18 2.44 Debian 13 Trixie ✅️ working ✅️ working
Clang 19 2.44 Debian 13 Trixie ✅️ working ✅️ working

On Debian it always work with GCC, whatever the binutils version.

On Ubuntu it stops working with GCC 8 on Ubuntu 20.04 Focal, with GCC 7 it works on Focal, but it was working with GCC 8 on 18.04 Bionic…

The patch always fixes the bug.

@illwieckz illwieckz changed the title nacl_helper_bootstrap: associate .rodata with :text PHDR, fix build on GCC nacl_helper_bootstrap: associate .rodata with :text PHDR, fix build with GCC Dec 12, 2025
@illwieckz illwieckz force-pushed the illwieckz/bootstrap-gcc branch from 4ba6efa to e24dc95 Compare December 12, 2025 07:00
@slipher
Copy link
Member

slipher commented Dec 12, 2025

The bug is triggered (at least on Arch Linux which I tested) when there is a note.gnu.property section. I guess this has something to do with the GCC configuration and is supposedly related to "branch protection" security features.

The broken binary has a faulty PT_LOAD command that overwrites the text section with zeroes. Definitely seems like a bug with ld.

Program Header:
    LOAD off    0x0000000000000000 vaddr 0x0000000000010000 paddr 0x0000000000010000 align 2**12
         filesz 0x0000000000001850 memsz 0x0000000000001850 flags r-x
    LOAD off    0x0000000000001000 vaddr 0x0000000000012000 paddr 0x0000000000012000 align 2**12
         filesz 0x0000000000000000 memsz 0x0000000000001008 flags rw-
    LOAD off    0x0000000000001850 vaddr 0x0000000000000000 paddr 0x0000000000000000 align 2**12
         filesz 0x0000000000000000 memsz 0x0000000000000000 flags ---
    LOAD off    0x0000000000002190 vaddr 0x0000000000010190 paddr 0x0000000000010190 align 2**12
         filesz 0x0000000000003e98 memsz 0x0000000000003e98 flags rw-
    NOTE off    0x00000000000001c0 vaddr 0x00000000000101c0 paddr 0x00000000000101c0 align 2**2
         filesz 0x0000000000000024 memsz 0x0000000000000024 flags r--
   STACK off    0x0000000000000000 vaddr 0x0000000000000000 paddr 0x0000000000000000 align 2**3
         filesz 0x0000000000000000 memsz 0x0000000000000000 flags rw-

What do you think of the approach in the slipher/orphan-discard branch? This simply discards all sections we didn't ask for, instead of letting the linker decide where to put them. A drawback could be that you lose debug symbols when building in debug mode (though that could be fixed if desired). It has the side effect of making the binary much smaller; not sure why since the unused sections aren't that big.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants