Skip to content

Raspberry Pi Setup

Jump to bottom
Jens Alfke edited this page Jul 8, 2019 · 6 revisions

As we move forward with support for Raspberry Pi it will be important to get a clear process in place so that we can reliably know how to generate a good build environment for it. This guide will show how to build a cross-compilation environment on a Mac which can compile and link a Raspberry Pi executable, without needing an actual Raspberry Pi device.

Part 1: Toolchain

You'll need a Mac, with a recent version of macOS and the unofficial Homebrew package manager. Install Homebrew if you haven't already. From Homebrew only two things are required:

The first is an appropriate binutils. This set of programs is what takes compiled code (object files) and converts them into actual machine language (in this case the end result is a standard binary format used across Linux distros called "ELF"). It is already likely installed on your machine, but in the x64 variant. Since we are cross compiling we need to install the variant that Rapsberry Pi expects: ARM with the GNU EABI, using hardware floating point (aka arm-linux-gnueabihf).

The second is a version of llvm with lld included, since the ld linker cannot understand the --sysroot argument. This will be used later in step 4, and the binutils programs will no longer be needed.

To install these, once Homebrew itself is installed, run:

brew install arm-linux-gnueabihf-binutils llvm

Part 2: Sysroot

NOTE: If you already have a sysroot generated you can skip this part.

You will need Docker installed in order to generate a Raspberry Pi (Raspbian) sysroot.

(What is a sysroot? To put it in easy terms: It means that you will set a directory and tell the compiler "this is the only place you are allowed to look for headers and libraries." This will effectively trick the compiler into thinking it is running in an actual Raspberry Pi environment. The sysroot contains subdirectories like usr/include and lib/arm-linux-gnueabihf that correspond to the ones found at the root of an actual Linux filesystem.)

To generate a sysroot, the high level steps are to

  1. obtain a disk image of Raspbian,
  2. modify it as necessary, and
  3. copy out the necessary libraries and C/C++ header files.

Get Raspbian

In order to get the largest support matrix, it is important to select as old of a version of Raspbian as possible. This is because of the versioning of the GNU C Library (which, in the end, everything in the world ends up using.) As a runtime requirement, a program must be compiled with the same version or older of the C library that is present on the system (i.e. a program compiled against glibc 2.24 will run on a system with 2.25 present, but not 2.23.) So we want to compile against the oldest compatible C library we can.

First, let's see what is available. If you need to use server / graphical libraries you will need the full version of Raspbian, but for our purposes we will use the Lite version which is only 1/3 of the size. A list of such images is available from the Raspberry Pi downloads server. Today I will choose the oldest version that is based on the current (as of July 2019) stable version of Debian (called "Stretch"). This is raspbian_lite-2017-08-17. You can download via HTTP or via torrent. Torrent is very much recommended as the HTTP server does not serve files with any speed.

After downloading, what you will have is a filed called 2017-08-16-raspbian-stretch-lite.zip. Save this file somewhere you recognize and start up a privileged docker ubuntu container:

docker run --privileged -it ubuntu:18.04 

# Once inside the Docker container:
mkdir raspbian
apt-get update
apt-get install -y unzip

Then, in another terminal outside of the docker container, copy the zip file to the container. This requires getting the docker container's ID first.

# In a macOS shell:
$ docker container ls

CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
364308b29218        ubuntu:18.04        "/bin/bash"         About an hour ago   Up About an hour    

# Note the container ID above
docker cp 2017-08-16-raspbian-stretch-lite.zip 364308b29218:raspbian/raspbian.zip

Now, perform the rest of the steps inside of the container (back in the other terminal). First unzip the image:

# In the Docker container:
cd raspbian
unzip raspbian.zip

Now what you will have is a disk image containing the filesystem of Raspbian called 2017-08-16-raspbian-stretch-lite.img. This needs to be mounted read-write (which is something that is not reliably possible on macOS due to the filesystem being in the extv4 format) and tweaked. Perform the following steps:

# In the Docker container:

# Install needed tools
apt-get install -y kpartx parted e2fsck-static \
    qemu qemu-user-static binfmt-support xz-utils rsync

# extend raspbian image by 128MB
dd if=/dev/zero bs=1M count=128 >> 2017-08-16-raspbian-stretch-lite.img

# set up image as loop device
# You will see output as in the following:
# add map loop0p1 (253:4): 0 85623 linear 7:4 8192
# add map loop0p2 (253:5): 0 11588248 linear 7:4 94208
# Remember which "loop" gets assigned, in this case loop0
kpartx -v -a 2017-08-16-raspbian-stretch-lite.img

# Use the same loop from before
parted /dev/loop0
    resizepart 2 -1s # Expand partition 2 to fill the space extended above
    quit

# Recreate the loop device, be sure to run both of these
# so the device actually gets removed (Docker is funny?)
# confirm the output still shows loop0
kpartx -d /dev/loop0
losetup -d /dev/loop0
kpartx -v -a 2017-08-16-raspbian-stretch-lite.img

# check file system
e2fsck -f /dev/mapper/loop0p2

# expand partition filesystem
resize2fs /dev/mapper/loop0p2

# mount at last
mkdir /mnt/raspbian
mount -o rw /dev/mapper/loop0p2  /mnt/raspbian
mount -o rw /dev/mapper/loop0p1 /mnt/raspbian/boot 

# optional
mount --bind /dev /mnt/raspbian/dev/
mount --bind /sys /mnt/raspbian/sys/
mount --bind /proc /mnt/raspbian/proc/

# optional, but recommended for part 4
mount --bind /dev/pts /mnt/raspbian/dev/pts

Modify Raspbian

Now what you have is the Raspberry Pi filesystem mounted into the directory /mnt/raspbian. This is where we are about to change our root to, and so there is one problem standing in the way. The ld.so.preload file on the filesystem contains libraries with absolute paths from the current root. However, none of them are needed for our purposes so we can just comment out every line:

# In the Docker container:
sed -i 's/^/#/g' /mnt/raspbian/etc/ld.so.preload

Because we installed qemu-user-static earlier, our system now has the ability to run executables of another architecture inline so long as they are statically linked, and only take dynamic links against certain system libraries. Combined with chroot this has the effect of turning the docker container into a makeshift Raspberry Pi emulator. If you want to confirm this for yourself, run this before the next step:

$ uname -m
x86_64

Now it's time for the magic command! Run chroot to change the system root to the Raspberry Pi filesystem:

$ chroot /mnt/raspbian/ /bin/bash

# Now try running uname again
$ uname -m
armv7l

Now we are emulating a Raspberry Pi environment, and so you can change it as you see fit and the image file that you copied way at the beginning will be updated to match. (However, we won't use the image file itself anymore.)

For building LiteCore, we need to install libicu-dev, so let's do that:

apt-get update
apt-get install -y libicu-dev

(Notice the "armhf" entries being installed, as apt-get automatically pulls the correct architecture into the correct place on the filesystem.)

Once that is done, simply exit out of the root, back to x64 land:

exit

Extract the libraries and headers

It's time to extract the sysroot from the filesystem. To do this we will use rsync on a few key directories, and make a compressed tar of the result.

mkdir /raspbian/sysroot
cd /raspbian/sysroot
rsync -rzLR --safe-links \
    /mnt/raspbian/./usr/lib/arm-linux-gnueabihf/ \
    /mnt/raspbian/./usr/lib/gcc/arm-linux-gnueabihf/ \
    /mnt/raspbian/./usr/include/ \
    /mnt/raspbian/./lib/arm-linux-gnueabihf/ \
    .

# The funny "dot" syntax above only copies from /usr and /lib
# as the "root" so the current folder will have those two
# folders
tar cfJv sysroot.tar.xz *

This will take a few minutes, but in the end the file will be 100 - 125 MB. You will get a bunch of messages about "symlink has no referent" and it is important to make note of these. These are all symlinks from /usr/lib to /lib that rsync is not able to handle because they are pointing to absolute paths outside of the new directory tree. I haven't found a way to deal with this yet, but on the host systems these symlinks will need to be recreated or else bad builds will happen.

Copy the sysroot to the host filesystem

Now, outside the container in the other session, copy the compressed tar out of the container:

# In the macOS shell:
docker cp 364308b29218:raspbian/sysroot/sysroot.tar.xz sysroot.tar.xz

Now extract the tarball somewhere (e.g. $HOME/sysroot). Finally we need to fix up the broken symlinks:

For each of the failed symlink entries:

  1. In the container: Find out what it points to (readlink <path that was in the message>)
  2. In the sysroot on the host machine: make a symbolic link using the relative path (cd parent/dir/of/symlink && ln -s ../../../lib/arm-linux-gnueabihf/<name from step 1> <name of symlink>)

This is a tedious process, but it makes the sysroot portable and you may now reuse this sysroot and will not have to repeat this step unless you need a new version of it. If you fail to do this step you will be met with some very odd linking and/or runtime errors because what will end up happening is since a static version of the same library exists in the same location as you are making these symlinks, these libraries will get statically linked into each unit instead of dynamically linked to all of them. This especially causes problems with libpthread and libc since these two libraries are so tightly coupled.

Now the environment is ready for cross compilation.

Cleanup

Finally we clean up the Docker image by reverting the ld.so.preload fix and unmounting the Raspbian disk image. (Note that the loop devices persist after the Docker image is destroyed, so this is important,)

# In the Docker shell:
sed -i 's/^#//g' /mnt/raspbian/etc/ld.so.preload
umount /mnt/raspbian/{dev/pts,dev,sys,proc,boot,}
kpartx -d /dev/loop0
losetup -d /dev/loop0

Part 3: Cross Compiling

clang has the ability to cross compile built into it already and the version that ships with the Apple Developer Tools appears to have the ARM targets built in. (This doesn't go all the way to using LLVM's libc++, but that will come in the next section.) For now, clang will handle the compilation and make use of GNU's linker and standard library to finish the job. However, the version of ld that ships with macOS seems to be a bit behind the times and doesn't understand some of the arguments. Luckily, clang has the ability to switch linkers. Unfortunately, it will only search for predefined names in the system path, and so the path needs to be tweaked to include /usr/local/opt/arm-linux-gnueabihf-binutils/arm-linux-gnueabihf/bin.

So go ahead and make a file called hello.cpp with some arbitrary contents:

#include <iostream>

int main()
{
    std::cout << "Hello" << std::endl;
    return 0;
}

Then run the following uber-command to cross-compile it:

PATH=/usr/local/opt/arm-linux-gnueabihf-binutils/arm-linux-gnueabihf/bin:$PATH \
    clang++ --target=arm-linux-gnueabihf \
            --sysroot=`pwd` \
            -isystem `pwd`/usr/include/c++/6.3.0 \
            -isystem `pwd`/usr/include/arm-linux-gnueabihf/c++/6.3.0 \
            -L`pwd`/usr/lib/gcc/arm-linux-gnueabihf/6.3.0 \
            -B`pwd`/usr/lib/gcc/arm-linux-gnueabihf/6.3.0 \
            --gcc-toolchain=`brew --prefix arm-linux-gnueabihf-binutils` \
            -fuse-ld=gold \
            -o hello \
            hello.cpp

There is a lot going on here, so let's examine what is going on. First, the PATH is being temporarily prepended to so that ld.gold can be found. Then, clang is being instructed via the --target argument to compile to the correct architecture. The next argument --sysroot instructs the compiler (and linker) to use the supplied directory as the root of the environment so that it only picks up libraries from there. The next couple -isystem additions add system include directories. The -L adds a library search path so that things like libstdc++ can be found. The -B adds a search path for the linker so it can find some fundamental things (like crtbegin.o and crtend.o) that go into every program that uses C. The --gcc-toolchain argument, which is dubiously named, provides the location of the utilities for creating actual ELF executables (ar, ld, etc). -fuse-ld instructs clang to call ld.gold instead of just ld and finally the -o just names the output executable. Note that you might need to adjust the 6.3.0 entry based on what version of Raspbian you created the sysroot from (e.g. Buster is 8 instead of 6.3.0)

Now you will have an executable called hello, which you can verify is an ELF executable:

$ file hello                                                         
hello: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-, for GNU/Linux 3.2.0, not stripped

If you upload this to a raspberry pi it will run and print "Hello", and if you examine it, you will see it is using GCC's standard library:

$ ldd hello
    linux-vdso.so.1 (0x7eda1000)
	/usr/lib/arm-linux-gnueabihf/libarmmem-${PLATFORM}.so => /usr/lib/arm-linux-gnueabihf/libarmmem-v7l.so (0x76f02000)
	libstdc++.so.6 => /usr/lib/arm-linux-gnueabihf/libstdc++.so.6 (0x76dbb000)
	libgcc_s.so.1 => /lib/arm-linux-gnueabihf/libgcc_s.so.1 (0x76d8e000)
	libc.so.6 => /lib/arm-linux-gnueabihf/libc.so.6 (0x76c40000)
	libm.so.6 => /lib/arm-linux-gnueabihf/libm.so.6 (0x76bbe000)
	/lib/ld-linux-armhf.so.3 (0x76f17000)

Appendix: Onward to libc++

LiteCore does build and run with GCC and its standard library, but for optimum compatibility we'd like to use Clang and libc++ as we do in our other Couchbase Lite builds. This requires a bit more work...

The current stable version of Debian is Debian 9 (Stretch). However, this version ships with very old versions of both clang and libc++ (3.8 and 3.5.2 respectively). The current testing version (Debian 10 Buster) has made a nice upgrade to the 7.0 release for both. So the only way to do this reliably is to target Buster only. Building libc++ and libc++abi from source for ARM is required for earlier versions. There are some issues doing this with 7.0, and the version of libunwind (which is a requirement on ARM) in 8.0 appears to have some kind of bug that causes rampant segfaults. So in order to use libc++, you need to repeat the sysroot generation process with a version that contains "buster" in the name (the newest one will do), but in the section where you run apt-get install -y libicu-dev instead run apt-get install -y libicu-dev libc++-dev libc++abi-dev. Then when you get to the rsync step, add another directory in before the . at the end: /mnt/raspbian/./usr/lib/llvm-7/. After that sysroot is set up, the new command to cross compile is as follows:

PATH=/usr/local/Cellar/llvm/8.0.0_1/bin:$PATH \
    clang++ -std=c++11 -stdlib=libc++ \
            --target=arm-linux-gnueabihf \
            --sysroot=`pwd` \
            -isystem `pwd`/usr/lib/llvm-7/include/c++ \
            -L`pwd`/usr/lib/gcc/arm-linux-gnueabihf/8 \
            -L`pwd`/usr/lib/llvm-7/lib \
            -B`pwd`/usr/lib/gcc/arm-linux-gnueabihf/8 \
            -fuse-ld=lld -o hello \
            hello.cpp

Note the changes to the invocation. std=c++11 was added to use the C++ 11 version of the language, and stdlib=libc++ to instruct the compiler to add linker flags to libc++ instead of libstdc++ for its standard library. The two -isystem entries were replaced with just one for the location of the libc++ headers. An -L entry was added to provide the location of the libc++ library (Note the existing one is still required for libgcc_s). The --gcc-toolchain argument was also removed as this is no longer needed, and finally -fuse-ld was updated to use lld instead of gold. This executable will run on a Debian 10+ based Pi, and examining shows that it has replaced GCC's standard library with LLVM's:

$ ldd hello
    linux-vdso.so.1 (0x7ee04000)
	/usr/lib/arm-linux-gnueabihf/libarmmem-${PLATFORM}.so => /usr/lib/arm-linux-gnueabihf/libarmmem-v7l.so (0x76ee5000)
	libc++.so.1 => /usr/lib/arm-linux-gnueabihf/libc++.so.1 (0x76e32000)
	libc++abi.so.1 => /usr/lib/arm-linux-gnueabihf/libc++abi.so.1 (0x76df8000)
	libgcc_s.so.1 => /lib/arm-linux-gnueabihf/libgcc_s.so.1 (0x76dcb000)
	libc.so.6 => /lib/arm-linux-gnueabihf/libc.so.6 (0x76c7d000)
	libatomic.so.1 => /usr/lib/arm-linux-gnueabihf/libatomic.so.1 (0x76c64000)
	libpthread.so.0 => /lib/arm-linux-gnueabihf/libpthread.so.0 (0x76c3a000)
	/lib/ld-linux-armhf.so.3 (0x76efa000)
	libm.so.6 => /lib/arm-linux-gnueabihf/libm.so.6 (0x76bb8000)
	librt.so.1 => /lib/arm-linux-gnueabihf/librt.so.1 (0x76ba1000)

Clone this wiki locally