Skip to content

Raspberry Pi Setup

Jump to bottom
Jim Borden edited this page Oct 17, 2019 · 6 revisions

As we move forward with support for Raspberry Pi it will be important to get a clear process in place so that we can reliably know how to generate a good build environment for it. This guide will show how to build a cross-compilation environment on a Mac which can compile and link a Raspberry Pi executable, without needing an actual Raspberry Pi device.

Part 1: Toolchain

You'll need a Mac, with a recent version of macOS and the unofficial Homebrew package manager. Install Homebrew if you haven't already. From Homebrew only two things are required:

The first is an appropriate binutils. This set of programs is what takes compiled code (object files) and converts them into actual machine language (in this case the end result is a standard binary format used across Linux distros called "ELF"). It is already likely installed on your machine, but in the x64 variant. Since we are cross compiling we need to install the variant that Rapsberry Pi expects: ARM with the GNU EABI, using hardware floating point (aka arm-linux-gnueabihf).

The second is a vanilla build of llvm. The one that comes installed on macOS is not equipped to handle using a sysroot when cross compiling.

To install these, once Homebrew itself is installed, run:

brew install arm-linux-gnueabihf-binutils llvm

Part 2: Sysroot

NOTE: If you already have a sysroot generated you can skip this part.

You will need Docker installed in order to generate a Raspberry Pi (Raspbian) sysroot.

(What is a sysroot? To put it in easy terms: It means that you will set a directory and tell the compiler "this is the only place you are allowed to look for headers and libraries." This will effectively trick the compiler into thinking it is running in an actual Raspberry Pi environment. The sysroot contains subdirectories like usr/include and lib/arm-linux-gnueabihf that correspond to the ones found at the root of an actual Linux filesystem.)

To generate a sysroot, the high level steps are to

  1. obtain a disk image of Raspbian,
  2. modify it as necessary, and
  3. copy out the necessary libraries and C/C++ header files.

Get Raspbian

In order to get the largest support matrix, it is important to select as old of a version of Raspbian as possible. This is because of the versioning of the GNU C Library (which, in the end, everything in the world ends up using.) As a runtime requirement, a program must be compiled with the same version or older of the C library that is present on the system (i.e. a program compiled against glibc 2.24 will run on a system with 2.25 present, but not 2.23.) So we want to compile against the oldest compatible C library we can.

First, let's see what is available. If you need to use server / graphical libraries you will need the full version of Raspbian, but for our purposes we will use the Lite version which is only 1/3 of the size. A list of such images is available from the Raspberry Pi downloads server. Since I'd prefer an image with libc++ already available, I'll use the newest one which currently is raspbian_lite-2019-06-24. You can download via HTTP or via torrent. Torrent is very much recommended as the HTTP server does not serve files with any speed.

After downloading, what you will have is a filed called 2019-06-20-raspbian-buster-lite.zip. Save this file somewhere you recognize and start up a privileged docker ubuntu container:

docker run --privileged -it ubuntu:18.04 

# Once inside the Docker container:
mkdir raspbian
apt-get update
apt-get install -y unzip

Then, in another terminal outside of the docker container, copy the zip file to the container. This requires getting the docker container's ID first.

# In a macOS shell:
$ docker container ls

CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
364308b29218        ubuntu:18.04        "/bin/bash"         About an hour ago   Up About an hour    

# Note the container ID above
docker cp 2019-06-20-raspbian-buster-lite.zip 364308b29218:raspbian/raspbian.zip

Now, perform the rest of the steps inside of the container (back in the other terminal). First unzip the image:

# In the Docker container:
cd raspbian
unzip raspbian.zip

Now what you will have is a disk image containing the filesystem of Raspbian called 2019-06-20-raspbian-buster-lite.img. This needs to be mounted read-write (which is something that is not reliably possible on macOS due to the filesystem being in the extv4 format) and tweaked. Perform the following steps:

# In the Docker container:

# Install needed tools
apt-get install -y kpartx parted e2fsck-static \
    qemu qemu-user-static binfmt-support xz-utils rsync

# extend raspbian image by 128MB
dd if=/dev/zero bs=1M count=128 >> 2019-06-20-raspbian-buster-lite.img

# set up image as loop device
# You will see output as in the following:
# add map loop0p1 (253:4): 0 85623 linear 7:4 8192
# add map loop0p2 (253:5): 0 11588248 linear 7:4 94208
# Remember which "loop" gets assigned, in this case loop0
kpartx -v -a 2019-06-20-raspbian-buster-lite.img

# Use the same loop from before
parted /dev/loop0
    resizepart 2 -1s # Expand partition 2 to fill the space extended above
    quit

# Recreate the loop device, be sure to run both of these
# so the device actually gets removed (Docker is funny?)
# confirm the output still shows loop0
kpartx -d /dev/loop0
losetup -d /dev/loop0
kpartx -v -a 2019-06-20-raspbian-buster-lite.img

# check file system
e2fsck -f /dev/mapper/loop0p2

# expand partition filesystem
resize2fs /dev/mapper/loop0p2

# mount at last
mkdir /mnt/raspbian
mount -o rw /dev/mapper/loop0p2  /mnt/raspbian
mount -o rw /dev/mapper/loop0p1 /mnt/raspbian/boot 

# optional, but recommended
mount --bind /dev/pts /mnt/raspbian/dev/pts

Modify Raspbian

Now what you have is the Raspberry Pi filesystem mounted into the directory /mnt/raspbian. This is where we are about to change our root to, and so there is one problem standing in the way. The ld.so.preload file on the filesystem contains libraries with absolute paths from the current root. However, none of them are needed for our purposes so we can just comment out every line:

# In the Docker container:
sed -i 's/^/#/g' /mnt/raspbian/etc/ld.so.preload

Because we installed qemu-user-static earlier, our system now has the ability to run executables of another architecture inline so long as they are statically linked, and only take dynamic links against certain system libraries. Combined with chroot this has the effect of turning the docker container into a makeshift Raspberry Pi emulator. If you want to confirm this for yourself, run this before the next step:

$ uname -m
x86_64

Now it's time for the magic command! Run chroot to change the system root to the Raspberry Pi filesystem:

$ chroot /mnt/raspbian/ /bin/bash

# Now try running uname again
$ uname -m
armv7l

Now we are emulating a Raspberry Pi environment, and so you can change it as you see fit and the image file that you copied way at the beginning will be updated to match. (However, we won't use the image file itself anymore.)

For building LiteCore, we need to install a few dependencies, so let's do that:

apt-get update
apt-get install -y libicu-dev libc++-dev libc++abi-dev

(Notice the "armhf" entries being installed, as apt-get automatically pulls the correct architecture into the correct place on the filesystem.)

Once that is done, simply exit out of the root, back to x64 land:

exit

Extract the libraries and headers

It's time to extract the sysroot from the filesystem. To do this we will use rsync on a few key directories, and make a compressed tar of the result.

mkdir /raspbian/sysroot
cd /raspbian/sysroot
rsync -rzLR --safe-links \
    /mnt/raspbian/./usr/lib/arm-linux-gnueabihf/ \
    /mnt/raspbian/./usr/lib/gcc/arm-linux-gnueabihf/ \
    /mnt/raspbian/./usr/include/ \
    /mnt/raspbian/./lib/arm-linux-gnueabihf/ \
    /mnt/raspbian/./usr/lib/llvm-7/ \
    .

# The funny "dot" syntax above only copies from /usr and /lib
# as the "root" so the current folder will have those two
# folders
tar cfJv sysroot.tar.xz *

This will take a few minutes, but in the end the file will be 100 - 125 MB. You will get a bunch of messages about "symlink has no referent" and it is important to make note of these. These are all symlinks from /usr/lib to /lib that rsync is not able to handle because they are pointing to absolute paths outside of the new directory tree. I haven't found a way to deal with this yet, but on the host systems these symlinks will need to be recreated or else bad builds will happen.

Copy the sysroot to the host filesystem

Now, outside the container in the other session, copy the compressed tar out of the container:

# In the macOS shell:
docker cp 364308b29218:raspbian/sysroot/sysroot.tar.xz sysroot.tar.xz

Now extract the tarball somewhere (e.g. $HOME/sysroot). Finally we need to fix up the broken symlinks:

For each of the failed symlink entries:

  1. In the container: Find out what it points to (readlink <path that was in the message>)
  2. In the sysroot on the host machine: make a symbolic link using the relative path (cd parent/dir/of/symlink && ln -s ../../../lib/arm-linux-gnueabihf/<name from step 1> <name of symlink>)

This is a tedious process, but it makes the sysroot portable and you may now reuse this sysroot and will not have to repeat this step unless you need a new version of it. If you fail to do this step you will be met with some very odd linking and/or runtime errors because what will end up happening is since a static version of the same library exists in the same location as you are making these symlinks, these libraries will get statically linked into each unit instead of dynamically linked to all of them. This especially causes problems with libpthread and libc since these two libraries are so tightly coupled.

Now the environment is ready for cross compilation.

Cleanup

Finally we clean up the Docker image by reverting the ld.so.preload fix and unmounting the Raspbian disk image. (Note that the loop devices persist after the Docker image is destroyed, so this is important,)

# In the Docker shell:
sed -i 's/^#//g' /mnt/raspbian/etc/ld.so.preload
umount /mnt/raspbian/{dev/pts,dev,sys,proc,boot,}
kpartx -d /dev/loop0
losetup -d /dev/loop0

Part 3: Cross Compiling

clang has the ability to cross compile built into it already and the version that ships with the Apple Developer Tools appears to have the ARM targets built in. (This doesn't go all the way to using LLVM's libc++, but that will come in the next section.) For now, clang will handle the compilation and make use of GNU's linker and standard library to finish the job. However, the version of ld that ships with macOS seems to be a bit behind the times and doesn't understand some of the arguments. Luckily, clang has the ability to switch linkers. Unfortunately, it will only search for predefined names in the system path, and so the path needs to be tweaked to include /usr/local/opt/arm-linux-gnueabihf-binutils/arm-linux-gnueabihf/bin.

So go ahead and make a file called hello.cpp with some arbitrary contents:

#include <iostream>

int main()
{
    std::cout << "Hello" << std::endl;
    return 0;
}

Then run the following uber-command to cross-compile it:

PATH=/usr/local/Cellar/llvm/8.0.0_1/bin:$PATH \
    clang++ -std=c++11 -stdlib=libc++ \
            --target=arm-linux-gnueabihf \
            --sysroot=`pwd` \
            -L`pwd`/usr/lib/llvm-7/lib \
            -Wl,-rpath-link,`pwd`/lib/arm-linux-gnueabihf \
            -Wl,-rpath-link,`pwd`/usr/lib/arm-linux-gnueabihf \
            -o hello \
            hello.cpp

There is a lot going on here, so let's examine what is going on. First, the PATH is being temporarily prepended to so that the proper clang++ can be found. After that the compiler is instructed to use libc++ instead of libstdc++. Then, clang is being instructed via the --target argument to compile to the correct architecture. The next argument --sysroot instructs the compiler (and linker) to use the supplied directory as the root of the environment so that it only picks up libraries from there. The -L adds a library search path so that libc++ and friends can be found. The -Wl,-rpath-link arguments set up linking paths for when an executable being built is looking up dependencies of its dependencies and finally the -o just names the output executable.

Now you will have an executable called hello, which you can verify is an ELF executable:

$ file hello                                                         
hello: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-, for GNU/Linux 3.2.0, not stripped

If you upload this to a raspberry pi it will run and print "Hello", and if you examine it, you will see it is using libc++.

$ ldd hello
	linux-vdso.so.1 (0x7ebe6000)
	/usr/lib/arm-linux-gnueabihf/libarmmem-${PLATFORM}.so => /usr/lib/arm-linux-gnueabihf/libarmmem-v7l.so (0x76f6f000)
	libc++.so.1 => /usr/lib/arm-linux-gnueabihf/libc++.so.1 (0x76ebc000)
	libc++abi.so.1 => /usr/lib/arm-linux-gnueabihf/libc++abi.so.1 (0x76e82000)
	libm.so.6 => /lib/arm-linux-gnueabihf/libm.so.6 (0x76e00000)
	libgcc_s.so.1 => /lib/arm-linux-gnueabihf/libgcc_s.so.1 (0x76dd3000)
	libc.so.6 => /lib/arm-linux-gnueabihf/libc.so.6 (0x76c85000)
	libatomic.so.1 => /usr/lib/arm-linux-gnueabihf/libatomic.so.1 (0x76c6c000)
	libpthread.so.0 => /lib/arm-linux-gnueabihf/libpthread.so.0 (0x76c42000)
	/lib/ld-linux-armhf.so.3 (0x76f84000)
	librt.so.1 => /lib/arm-linux-gnueabihf/librt.so.1 (0x76c2b000)

Appendix: Regarding libc++

LiteCore does build and run with GCC and its standard library, but for optimum compatibility we'd like to use Clang and libc++ as we do in our other Couchbase Lite builds. This requires a bit more work...

The current stable version of Debian is Debian 9 (Stretch). However, this version ships with very old versions of both clang and libc++ (3.8 and 3.5.2 respectively). The current testing version (Debian 10 Buster) has made a nice upgrade to the 7.0 release for both. So the only way to do this reliably is to target Buster only. Building libc++ and libc++abi from source for ARM is required for earlier versions. There are some issues doing this with 7.0, and the version of libunwind (which is a requirement on ARM) in 8.0 appears to have some kind of bug that causes rampant segfaults.

Clone this wiki locally