crosstool-NG HOWTO & gotchas

cross-toolchains are typically used to build code that is supposed to run on a machine with a different architecture, e.g. when developing embedded code for an ARM-based box on a x86 dev machine. however, they’re also very useful when you want to make sure that all the devs in a group use exactly the same set of toolchain binaries and libraries regardless of which versions of gcc, headers and libraries those devs have installed locally on their dev machines.

a cross-toolchain consists of a bunch of tools and libraries, which must play well together if it’s ever to be useful. coming up with a specific set of specific versions of tools/libs that will peacefully co-exist is not for the faint at heart, especially if you have wide-ranging requirements and want extra goodies (which you will want). it is even more of a PITA if you want to build a static (and relocatable) cross-toolchain.

crosstool-NG can ease the pain somewhat, though it will not make it go away totally. it’s a rewrite of the original crosstool by Yann E. Morin. for an old overview of this tool given by Yann at Embedded Linux Conference Europe 2009 see: video, slides.

obtaining crosstool-NG:

crosstool-NG is NOT in fedora’s yum repos (as of F15), so grab it from its home page, e.g.:

wget http://crosstool-ng.org/download/crosstool-ng/crosstool-ng-1.14.1.tar.bz2

to build crosstool-NG itself, you’ll need at least:

sudo yum install bison flex gperf

then the standard:

./configure --prefix=$HOME/tmp/ct-ng && make -j4 && make install

overview:

crosstool-NG (from here on “CT”) comes with a largish set of “Preconfigured toolchains”, or “samples”. those, allegedly, were proven to work together to one degree or another, and can provide inspiration (read: copy-paste) for creating your own. to see the list of such samples:

ct-ng list-samples

to see the components (tools/libs) that make up such a sample and their versions:

ct-ng show-<sample_name>

e.g. (an actual sample output):

ct-ng show-x86_64-unknown-linux-gnu
    x86_64-unknown-linux-gnu  [G  ]
        OS             : linux-2.6.33.20
        Companion libs : gmp-4.3.2 mpfr-2.4.2 ppl-0.10.2 cloog-ppl-0.15.9 libelf-0.8.13
        binutils       : binutils-2.20.1a
        C compiler     : gcc-4.4.3 (C,C++,Fortran,Java)
        C library      : glibc-2.9
        Tools          : dmalloc-5.5.2 duma-2_5_15 gdb-6.8a ltrace-0.5.3 strace-4.5.19

to see similar output for a toolchain you config yourself using menuconfig:

ct-ng show-config

after a number of false-starts, i arrived at the following mix of tools/libs versions that are close to my requirements and can be built into a static cross-toolchain on fedora-15, and seem to pass initial tests:

x86_64-acme-linux-gnu  [l X]
    OS             : linux-3.0.18
    Companion libs : gmp-4.3.2 mpfr-3.1.0 ppl-0.10.2 cloog-ppl-0.15.10 mpc-0.9 libelf-0.8.13
    binutils       : binutils-2.21.53
    C compiler     : gcc-4.5.3 (C,C++)
    C library      : glibc-2.13
    Tools          : dmalloc-5.5.2 duma-2_5_15 gdb-7.3a ltrace-0.5.3 strace-4.6

resulting disk usage is around 6GB, spread as follows:

~5.5GB  $CT_ROOT/.build
~0.5GB  $CT_XTOOLS/x86_64-acme-linux-gnu

building the cross-toolchain:

dir names used here:

$CT_INST - this is the --prefix you specified during ./configure, where your installed CT will land, assuming non-root install.
$CT_ROOT - this is wherever you run ct-ng binary from. CT will do everything under that dir, so make sure to run ct-ng inside a dedicated dir.
$CT_XTOOLS - root of the resultant cross-toolchains’ prefix dirs, this is where the final toolchains land.

flip through the dialogs of:

ct-ng menuconfig

choosing the packages you want. for best results you’ll prolly want to enable the EXPERIMENTAL stuff early on, otherwise most of the recent package versions will not be available and a lot of useful cross-toolchain functionality will be missing.

first things first:

“Paths and misc options” ->
- “Prefix directory” - this is where the final cross-toolchain will land, called $CT_XTOOLS in this document. it defaults under your $HOME/x-tools, change as necessary.
- “Number of parallel jobs” - set according to your number of cores/CPUs, though help recommends to set it to TWICE the number of CPUs in my experience that’s way to high.
- “Maximum allowed load” - very handy, again, set according to the HW you build on, leaving yourself some breathing space.
“Toolchain option” ->
- “Toolchain ID string” - shows up on gcc --version output as: acme-gcc (crosstool-NG 1.14.1 - <ID_string>) 4.5.3. the author recommends to put there date or build number.
- “Tuple’s vendor string” - the default is unknown. “target tuple” is the combo: arch-VENDOR-kernel-system, e.g. powerpc-e300c3-linux-gnu. this is the VENDOR bit of the tuple.
- “Tuple’s alias” - this one’s handy, it generates short-named symlinks to the same tools right alongside the tools themselves. e.g. if you set it to acme and build powerpc-e300c3-linux-gnu tuple you’ll end up with powerpc-e300c3-linux-gnu-gcc AND acme-gcc binaries under bin.

the rest - per your target and requirements.

if you’re even slightly adventurous in your configuration, you’ll witness a pile of failures before you make the darned thing build your cross-toolchain to your liking, most of them trivial and obvious, some - well documented. make sure to read:

$CT_INST/share/doc/crosstool-ng/ct-ng.1.14.1/B\ -\ Known\ issues.txt

then see “known issues” at the bottom of this document.

first step in troubleshooting failed build is to open $CT_ROOT/build.log, jump to the bottom and try to guess what went wrong. if it’s some failed dependency that one of the packages’ ./configure script notices, you may have to drill down to the specific config log to figure out what exactly went wrong and made ./configure say that some thing or other is “missing”. per-tool/lib logs land under:

$CT_ROOT/.build/<toolchain_name>/build/build-<package>/config.log

e.g.:

/tmp/x86_64-acme-linux-gnu/build/build-ppl/config.log

NOTE: CT sporadically fails to download packages from sourceforge, even though they appear to be there. just peek at the log that gets created as:
$CT_ROOT/build.log
look up which package CT tried to download and failed, copy the address and download it manually into:
$CT_ROOT/.build/tarballs
CT will notice the packages already there and use them when you re-run it. strace & duma are especially prone to this at this time of the year…

if still in the experimentation stage, you’ll want to make sure you:

enable [EXTRA] traces to see at a glance where CT fails. you’ll want “Paths and misc options” -> “Maximum log level to see” set to EXTRA for that.
enable “Paths and misc options” -> “Debug crosstool-NG” and then under it: “Save intermediate steps” (“gzip saved states” is up to you). rest assured your build WILL fail due to dependencies for a while before you nail it. unless you have this option on, when you’ll restart the build again after fixing issues, CT will do everything FROM SCRATCH. with this option on, after each tool/lib successfully built, CT will print (with tool/lib name instead of <step-name>, of course):
```
Saving state to restart at step '<step-name>'...
```
and you’ll be able to restart from that specific LNG state by doing:
```
ct-ng <step-name>+
```
e.g.:
```
ct-ng libc+
```
YOU WANT THIS OPTION ON UNTIL YOU FIGURE OUT YOUR PERFECT SET OF TOOL/LIB VERSIONS THAT WORKS. no, really, you do!
back up the tarballs that CT downloads in the process to the:
```
$CT_ROOT/.build/tarballs/
```
dir - they DO get wiped out in case of ct-ng distclean and CT WILL take ages to re-download the entire >200MB again! stash them someplace read- only, then specify this location under Paths and misc options” -> “Local tarballs directory”.

read the help messages of various options.

you prolly will want to build a fully static cross-toolchain to make sure it’ll run on any reasonably similar system (e.g. fedoras/RHELs/CentOSes) and can be “deployed” by a simple untar without having to install any libs on target or specify any paths to any .so libs (or just dumped on a central server exporting NFS to be mounted from the dev machines - though you’ll pay dearly in build times for that stunt!). to build even a moderately useful static cross-toolchain you’ll probably need to install at least:

sudo yum install glibc-static zlib-static libstdc++-static ncurses-static

you can wait for build to fail case-by-case until it forces you to install those, or just take my word for it and install them beforehand.

NOTE: recent versions of GDB require expat (WTF?), static cross-gdb requires static libexpat.a, but for some reason, the expat package on fedora does NOT provide a static lib, and unlike the above *-static packages there’s NO expat-static package on fedora (but there’s an open bug in RH’s bugzilla, if that’s any consolation). although CT downloads expat tarball, it ONLY builds a static version for the target, and even then, during native gdb phase which comes AFTER the cross-gdb phase. Yann refuses to build the libexpat.a for HOST for ideological reasons so, just ./configure + make the same version of expat as CT grabs (just reuse the same tar) and park the resultant libexpat.a under /usr/lib64. GRRR!!!

sanity-checking your cross-toolchain:

once you’ve managed to build your cross-toolchain start-to-finish without failures, it’s HIGHLY RECOMMENDED that you do some sanity checks on it. tick off: “Companion libraries” -> “Check the companion libraries builds” and “Test suite” -> “GCC test suite” in menuconfig, then re-start the build from scratch. no need for distclean, but do a ct-ng build - restarting from the middle using <stage>+ will NOT pick up the changes in .config!

the former option will run the tests during the build, so once the build is successfully over - you’re good. it also takes oh-shit-long (on the order of 90 minutes on a 4 x 3.1GHz machine with gobs of RAM) which is why you DON’T want to enable it by default…

the latter option just builds the GCC test suite, leaving it under:

$CT_XTOOLS/<toolchain_name>/test-suite/gcc/

this one is a bit of a beast: it’s really designed to TEST that a cross-compiler works by running compiled test snippets on an actual target. it’ll need dejagnu + expect on the host, and passwordless ssh connection to a target. IF your host is able to run the target binaries, like a 64-bit -> 32-bit cross-compiler for the same arch, you can just specify localhost as a target. luckily, i THINK CT pre-configures the test suite by putting the right strings into its Makefile, so all you really need to do is:

sudo yum install dejagnu

then kick off the tests (note DG_TOOLNAME is just gcc vs g++!):

make DG_TOOLNAME=gcc DG_TARGET_HOSTNAME=<tgt_addy> DG_TARGET_USERNAME=<user>

e.g.:

make DG_TOOLNAME=gcc DG_TARGET_HOSTNAME=127.0.0.1 DG_TARGET_USERNAME=luser

(where luser is the username dejagnu will use to connect, q.v. passwordless above). this will first compile a bunch of compile-only GCC regression tests on your host, then enter a loop where on each iteration it’ll compile a single compile+run regression test, scp it onto <tgt_addy> (possibly prompting you with yes/no if it’s the 1st ssh connection to that host, so don’t leave unattended if you want results!), then it’ll ssh to <tgt_addy> to run this test, etc. finally, repeat for g++:

make DG_TOOLNAME=g++ DG_TARGET_HOSTNAME=127.0.0.1 DG_TARGET_USERNAME=luser

the whole process took some 3.5 hours for gcc + 40min for g++.

there’s a pretty good chance that in the tests summary you WILL see SOME “unexpected failures” (likely the same several tests failing multiple times with different compilation flags, inflating the reported failures numbers). google up the circumstances on a per case basis: usually it boils down to bugs already reported upstream in RH/GNU/sourceware bugzillas, some of them already fixed upstream and possibly released in newer tools versions. MAKE SURE THAT THIS IS THE CASE and that it’s NOT your particular cross-toolchain that is broken, or you’ll find yourself debugging ghosts during actual development using the resultant cross-toolchain!

for the sake of comparison, on my cross-toolchains i got results in the vicinity of >20 “unexpected failures” for gcc, one or more “unexpected failures” for g++, the latter often involving the linker.

known issues:

older glibc refuse to build with make v3.82: “mixed implicit and normal rules”. just select “Companion tools” -> “Build some companion tools” and select make which will build make v3.81. CT will use it happily from there on.
ppl turned out to be a spiteful little thing, bearing a grudge against GMP (and CT users, apparently):
```
[INFO ]  Installing PPL
[ERROR]    configure: error: Cannot find GMP version 4.1.3 or higher.
```
PPL v0.11* configure script will ignore pretty much ANY version of GMP you might select in CT config. just fall back to v0.10*: gmp-4.3.2 + ppl-0.10.2 seem to work, but will probably necessitate downgrade of GCC from v4.6.x to v4.5.x.
gold,ld and ld,gold for linkers will FAIL to build static toolchain:
```
[EXTRA]    Building binutils
[ERROR]    g++: error: unrecognized option '-all-static'
```
-all-static is a libtool flag, gold doesn’t use libtool, so - yes, you’re screwed. stick to ld-only for linker for static cross-toolchains.
static cross-gdb fails to build, whining about expat:
```
[EXTRA]    Building cross-gdb
[ERROR]    configure: error: expat is missing or unusable
```
even though you have expat RPMs installed - see above, fedora don’t bundle the static libexpat.a anymore, so you’ll need to build it yourself.

gdbserver build barfs:

[ALL  ]    /home/luser/tmp/x-tools/x86_64-acme-linux-gnu/lib/gcc/\
    x86_64-acme-linux-gnu/4.5.3/../../../../x86_64-acme-linux-gnu/bin/ld:\
    /home/luser/tmp/x-tools/x86_64-acme-linux-gnu/lib/gcc/\
    x86_64-acme-linux-gnu/4.5.3/crtbeginT.o: relocation R_X86_64_32\
    against `__DTOR_END__' can not be used when making a shared object;\
    recompile with -fPIC
[ALL  ]    /home/luser/tmp/x-tools/x86_64-acme-linux-gnu/lib/gcc/\
    x86_64-acme-linux-gnu/4.5.3/crtbeginT.o: could not read symbols: Bad value
[ALL  ]    collect2: ld returned 1 exit status
[ERROR]    make[1]: *** [libinproctrace.so] Error 1
[ALL  ]    make[1]: *** Waiting for unfinished jobs....
[ALL  ]    make[1]: Leaving directory `/home/luser/tmp/crosstool-acme/.build/\
    x86_64-acme-linux-gnu/build/build-gdb-gdbserver'

no known solution, but it’s been noted before. in the meanwhile - just drop gdbserver.