Cross Compiling 101

In this blog post, I hope to provide some introductory stepping stones to cross-compiling and working with embedded systems. Use this post as a guide for themes to research and get you started on your way. I am by no means an embedded expert, but I have learned a great deal during my time working with a few different embedded systems.

Motivation
Why does a roboticist need embedded systems? These days mobile chips are becoming powerful enough to run a variety of algorithms, from control to sensor analysis to communication. Some mobile boards are beginning to offer dedicated GPUs with support for GPGPU languages (CUDA and OpenCL). The advantages of these full systems is their light weight, decent power, and tiny power requirements. For our hands on, we will focus on SECO’s new Carma DevKit. This board has a quad core Tegra 3 along with a Quadro dedicated gpu. There are tons of other well supportedrry Pi’s and TI’s OMAP boards. Carma Kit

 

Standard Embedded System
Many roboticists have installed an OS (ubuntu, mint) and found it to be easy. With embedded systems, things are little more complicated in that you must compile a a kernel image for your specific system. That kernel image must be loaded by a boot loader with correct kernel arguments, which differs from our standard BIOS based PC boot loaders. Finally, we need to load the drivers(modules) we compiled against our kernel image. I am still learning this process myself, but lucky for us tons of documentation and tools exist on this subject. However, sometimes it helps to have a general overview of what we’re working with and how it works. So we’ll examine the basic steps behind setup.

The standard boot loader for embedded systems is U-Boot. This boot loader is small enough to fit on most boards memory, advanced enough to be compiled with basic drivers for our system and offers a configuration console. When booting, U-Boot allows a user to interrupt the boot process and configure the boot parameters. Boot parameters control a variety of settings for the system itself. We can set video memory for cards with gpu chips, enable or disable specific drivers, the device and memory location of our kernel image,  and/or the device and memory location of our root file system. Once the kernel image is loaded, the boot loader will attempt to load a root filesystem. There’s a few approaches to this, the most common I’ve seen are NFS (Network Filesystem) and SD card based.  With a NFS approach, tftp is used to transfer necessary files from a host computer to the board which is booting. This may include the kernel image itself! With an SD card, typically we use “dd” to specifically write the U-Boot binary in the correct memory locations (usually described in your systems manual), add the kernel image using dd, set its location in U-Boot and go from there. An indepth example of these approaches can be found on U-Boot‘s webpage. An even more indepth tutorial on U-Boot and embedded system setup can be found at this website.

Cross-Compiling Background
The basis of cross compiling is compiling for a TARGET machine on a HOST machine. Typically the TARGET machine will differ in platform than the HOST (for example, compiling for ARM on an x86). The HOST machine usually installs a prebuilt binary cross-compiler (you can also build it yourself). If installed through a package manager, the package manager will install some basic cross compiler libraries for you as well. You will also sometimes get cross-compilers with Board Support Packages.The combination of a cross compiler, linker, and associated libraries is called our toolchain. Many build environments will take advantage of toolchains to cross compile for you. For example autoconf and cmake will both cross compile for you.

Autoconf is tricky, in that the flags you use have been revamped to do what you want. You would use the –target option. However carefully examine the output of the script, it will tell you if it found your cross-compile toolchain. (It looks for [type]-gcc NOT [type]-gcc-4.5, so set up your softlinks!) CMake makes it extremely easy, you simply specify a toolchain file. A more indepth explanation can be found here as well as in CMake’s documentation.

The major difficulty with cross compilation is dealing with symbols and libraries. Our HOST compiler uses the typical ld search paths (/lib, /usr/lib, /usr/local/lib), but our cross-compiler needs cross-compiled libraries and symbols. When cross compiling, the difficulty really lies in cross linking, using the correct libraries for your TARGET. There are tons of tiny assumptions that can create errors which make no sense to someone entering the cross compiling world! We’re going to go over a few debugging techniques here as well as some important flags.

Cross Compiling Headaches
The most important thing in cross linking is to know what libraries are being used; to examine the search paths of your compiler you can use the -print-search-dirs:

arm-linux-gnueabi-gcc -print-search-dirs

Second, use linking flags to correctly specify linking behavior. You should already be aware of the -L flag’s use in specifying new search directories, but more importantly you should be aware of the -Wl flag. This flag allows you to pass flags to your linker, this is useful when you’re using a compiler that also calls the linker for you on the object files. There are tons of linker flags, but there are only two main ones you needs to know -rpath and -rpath-link. These two flags help specify the runtime path of your binary object, in other words where it should look for dynamic libraries it needs to load. The -rpath flag will help remove “undefined symbol” messages at runtime when you’ve obviously linked the library.

But wait, if I’m compiling a binary that runs on another system I can’t set the -rpath to the location of my toolchain, it won’t be right on the TARGET system! That’s correct, which is why -rpath-link exists. This flag allows us to tell the linker that we only wish to use this runtime path while linking and resolving dependencies, but use the standard runtime paths. For example, our libcudart.so is in /home/constantin/workspace/linux_tool_chain/lib on our HOST machine but in the standard /usr/local/lib on our TARGET machine. So our compile line would be:

arm-linux-gnueabi-g++ -o test_cuda test_functions.cpp 
  -L/home/constantin/workspace/linux_tool_chain/lib -lcudart

or, if we are linking a dynamic library cross compiled for our cross-compiled libcudart.so

arm-linux-gnueabi-g++ -o test_cuda test_functions.cpp 
  -L/home/constantin/workspace/linux_tool_chain/lib 
  -Wl,-rpath-link,/home/constantin/workspace/cross_compiled_opencv/lib 
  -lcudart -lopencv_core -lopencv_gpu

In order to link in our libopencv_gpu.so, which has symbols from libcudart.so, we need to tell it what library has the CUDA symbols and the runtime path of the library but ONLY at link time.

There are a few interesting libraries that you need to watch out for: libpthread.so and libc.so. These are no normal shared objects, they’re scripts (do a less on /lib/libc.so)! So when you get Board Support Packages and you link directly to the TARGET filesystem’s directory, you might end up with errors. This often occurs when you have an NFS filesystem and you link directly to the root filesystem which exists on the same harddrive as your cross compile environment. The scripts execute and they instead link with your HOST’s system libraries. A great explanation on this topic can be found at this Stack Overflow question.

Hopefully that’s enough to get you started or help you deal with some very obscure bugs! Next time, we’ll focus directly on the CARMA Seco Board, as well as how to compile using CMake.

Posted in:
About the Author

Constantin Savtchenko

I attended Lehigh University, focusing on algorithms, and graduated in 2011 with a Masters in Computer Science. At Neya I have been part of a variety of projects all differing in domain. I have a wide range of programming experience, including parallel programming with GPU's.