Gentoo x86_64/amd64 on x86_64/amd64 crossdev guide.

You got an underpowered amd64 machine on which you want to install Gentoo. But instead of waiting for it for days to compile libreoffice or chromium, you wish you could compile it on a faster desktop. But the faster desktop is binary incompatible with the slower machine -- it does not support instructions like that of the avx512 family, gfni, hle, pku, sgx, vaes, vpclmulqdq etc... etc... etc.... but you want those instructions on your underpowered machine because that's the Gentoo way, and underpowered machine is the one where you would like to have extra speed of -march=native.

NOTE: QT and dependent packages (like KDE) are headache. If you're building QT packages, don't bother with this setup. chrooting is better. Disable the extra instruction on a per package basis (package.env) my modifying CFLAGs with things like -mno-avx512f -mno-vpclmulqdq etc...

NOTE: cross-building support for gentoo ebuilds is like a 2nd class citizen. 1st class being building on the machine that you're using. Reason being there are hardly any users using crossdev and upstream hardly cares about crossbuilding. You can use distcc, but then there is no support for rust, java etc... and fails to compile remotely for various reasons. Also if you're an LTO user, then linking will be done on the slow machine and with LTO it take exceptionally long to link. For rust there is sccache also, but I could not get the distributed compiling to work even for firefox on Gentoo.

First you need to readup upon the these 2 guides before proceeding into this one -- 

https://wiki.gentoo.org/wiki/Embedded_Handbook/General/Creating_a_cross-compiler

https://wiki.gentoo.org/wiki/Crossdev

This setup deviates from these 2 guides by the fact that these guides are essentially incomplete. Because although they provide you the procedures to build a working Gentoo install, they're mum upon how to actually use it on the target machines and what to do when you want to update it after 'using' the gentoo OS on the target machine. Also gentoo setup that crossdev makes are not exactly 'independent' -- in a sense that they may not have a working toolchain and rely completely on the builder machine for building all the packages.

The 'unconventional' setup --

This setup creates completely independent Gentoo installs which can compile on their own (i.e. has a working toolchain).

The idea here is to copy over your working gentoo install (or maybe even stage3) to /usr/<archspec> (where <archspec> is like x86_64-<chost>-linux-gnu, i.e. of your choice), do operations on it like updating, installing packages etc... and transfer it to your actual machine so it can boot from it. Of course for building a new package, you don't need to transfer the /usr/<archspec> back and forth; you can use gentoo binpkg support for that. This way you can build a binary package on your powerful desktop and transfer the binary package to the other machine.

Now crossdev has some issues with this setup. It assumes that /usr/<archspec> is empty when you install a toolchain to /usr/<archspec> otherwise building it fails with some strange errors. Once the toolchain is installed, it works with no problem even if you mangle it's contents /usr/<archspec> with the weak machine's rootfs. What you can do to work around over this is instead of copying over your weak machine's rootfs to /usr/<archspec>, you mount --bind it from some other place after the crossdev's <archspec> toolchain has been built. Just ensure that if you ever built the weak machine's toolchain again, just umount /usr/<archspec>.

Because this is amd64 on amd64, the <chost> defaults to 'pc'. So when building your toolchain using crossdev, set <chost> to something which is NOT pc. For the same reason, in /usr/<archspec>, you've to also change the CHOST maybe even for your builder machine because -- it's your choice. You can follow this guide for that. you can do this in chroot; if this is a stage 3 tar archive, you can safely chroot into it because it's generic as of now.

Also it's critically important to have a common portage tree between the build host and /usr/<archspec>, otherwise expect various issues here and there.

On multilib platforms, you've to enable mulitlib on the cross-toolchain built by crossdev. See this for detail. This essentially implies you've to append -A 'amd64 x86' to the crossdev command line options and unmask the multilib USE flag of the generated crossdev toolchain packages before building the toolchain for /usr/<archspec>.

CBUILD/CHOST concepts --

Now let's dive into the concepts of CBUILD/CHOST which is a toolchain concept used in the industry; this is needed to fix failed builds.

These are common across configure scripts (autoconf), but portage will 'translate' their meanings to have a similar affect across different toolchains.
CHOST is passed to the --host in the configure script (which is a part of the source code's build system) which implies 'cross-compile to build programs to run on CHOST' – i.e. the binaries produced will be intended to be run on <archspec> (same definition as in crossdev). The possible values of CHOST is also defined as <archspec>. The effect of this will be that portage will execute commands of the toolchain (gcc/llvm etc..) prefixed with <archspec>-* -- all this because gentoo assumes that binaries for <archspec> can only be produced by this compiler. No, changing the CHOST to make gentoo use another compiler (for e.g. the host's compiler when cross compiling) is NOT a good idea since the CHOST is also reflected in certain package's installed files and it also affects where the files are installed (because some paths have the CHOST in them); however these are mostly packages which belong to the toolchain or are system packages.

CBUILD – This is passed to configure script as –build=${CBUILD}; this implies the machine on which the building is happening. If CBUILD != CHOST it tells the toolchain that it's cross compiling and it's going to take a different route and avoid executing built binaries. However I believe that it's affect is only limited to determining if cross compiling should be activated or not. If you are cross compiling and CBUILD=CHOST, the compilation may fail if the build binaries (which have CFLAGS compatible for CHOST) are executed by the toolchain (for whatsoever reason) as a part of the build process ('helper programs' for completing the compilation process) and the CBUILD machine (build host) is incapable of running binaries produced for CHOST.

For a self independent machine, set set CBUILD=CHOST. When cross compiling, ensure to set the 2 different, although sometimes you can resolve compilation issues by setting CBUILD to CHOST (and not the other way around to prevent confusing your OS).

Changes required for crossdev's emerge

Now let's prepare the unconventional /usr/<archspec> so crossdev can be used for it. All you have to do is export the ROOT and PORTAGE_CONFIGROOT and set it to /usr/<archspec>. As a precaution also set SYSROOT to the same value. Now you may use <archspec>-emerge. For QT packages, export QT_HOST_PATH=/ will also help.

When packages fail to build -- 

As I said before, crossbuilt support for packages is 2nd class. You can expect a LOT of failures, but because this is x86 on x86 there are multiple workarounds over the crossdev specific build failure.

Method 1 -- 

Realize that you can cross build using plane emerge instead of <archspec>-emerge. It requires the following environment variables to be set -- 

export ROOT=/usr/<archspec>
export PORTAGE_CONFIGROOT=/usr/<archspec>

export SYSROOT=/usr/<archspec>
export PKG_CONFIG_PATH=/usr/<archspec>/usr/lib64/pkgconfig
export PKG_CONFIG_SYSROOT=/usr/<archspec>
export QT_HOST_PATH=/

export CBUILD=<archspec of the build host like x86_64-pc-linux-gnu>

export CHOST=<archspec>

 

You may try to build the package with these environment variables exported instead of <archspec>-emerge to see if it builds successfully; although in 99% of the cases this is just like crossdev.

Method 2 -- 

If you can chroot, then try to build the package in chroot. QT package will fail however.

Method 3 --

You may explore cb-emerge (called 'crossboss')

Method 4 -- 

Swap cbuild and chost's values as set in method 1. This will make the source code's build system cross-compile BUT using the build host's toollchain.

Method 5 --

Make changes to method 1's exported environment variables. Change the value of CBUILD to CHOST, i.e. CBUILD=<archspec>

This will make the build system of the package assume that you are NOT cross compiling. Any build failures because of the incapability of the package's build system to cross-compile or a bug in it will not be triggered. However this may cause traps: XXXXXXX trap invalid opcode in which case you've to play around with package.env to disable the incompatible instruction.

Method 6 --

Make changes to method 1), set CHOST=CBUILD, i.e. CHOST will be set to <archspec>; this will make portage use the build host's toolchain.

Method 7 --

Only build using ROOT and PORTAGE_CONFIGROOT set.

Method 8 -- 

Install build time dependencies on both /usr/<archspec> and the build host. These are very common bugs among packages. Sometimes you need to install run time dependencies on the build host also for the build to complete successfully. This is especially true when you've been using hacks like setting CBUILD=CHOST.

Method 9 --

Ensure build time dependencies are are identical on the build host and /usr/<archspec>. This is specifically a problem with dev-libs/protobuf.

Method 9 -- 

Even if you're cross compiling normally with crossdev, you may receive traps: XXXXXXXX trap invalid opcode. Here you've to change your package.env to iron out the incompatible instructions.

Java packages -- 

Only build with ROOT and PORTAGE_CONFIGROOT set. Otherwise you have to resolve dependency hell manually.

Multiple GCC versions

In case you need multiple GCC version, you can do that too using crossdev. Just specify a separate GCC version and it'll install it to a separate slot. To select the gcc version use eselect gcc.

Building binary packages 

In case you've changed your CHOST for successfully building the package, in the Packages file in the PKGDIR will contain CHOST set to the build host which will prevent it from being installed on the the machine on which /usr/<archspec> is intended to be used. Therefore you've to modify the Packages file and change the CHOST to <archspec>.

LLVM/clang -- 

Unlike GCC where a gcc is build specification for a CPU architecture (sans the tuning of individual instructions) a single clang install can build binaries for multiple architecture while cross compiling. crossdev knows about this therefore it does not build a separate clang/lang toolchain. But when you are building for gcc, clang is unsupported anyway (it's never by crossdev), so in your gcc machine, if you want clang support for cross compiling, you need to create symlinks as such -- 

for i in `find /usr/lib/llvm/ -mount -name 'x86_64-<CHOST of build host>-linux-gnu-*'`; do ln -s `echo -n $i | grep -Po '[^/]+$'` `echo -n $i | grep -Po /.*/``echo -n $i | grep -Po '[^/]+$' | sed s/-<CHOST of build host>-/-<CHOST of /usr/<archspec>-/`; done

This is essentially going to search all clang/llvm related binaries in /usr/lib/llvm/ and create a symlink of <archspec> pointing to your build machine's binaries.

Gentoo benchmarks with vs without avx512(-mno-avx512)

So I had to migrate my crossdev setup which I was using to maintain my laptop to a chroot-maintained setup on my non-avx512 capable workstation. In order to do this, I had to disable avx512 for the whole laptop rootfs and rebuild gentoo; in the process I also updated Gentoo so it was not exactly an apples to apples comparison; but it shouldn't matter much because there was very less difference in the package versions.

XZ benchmarks (lower the better) -- 

Bash benchmarks (lower the better) -- 

Compare and ffmpeg (lower the better) -- 

 

OpenSSL -- 

The performance difference between compare could be seen between Debian vs gentoo benchmarks too. So this not a mistake.

This machine is an icelake-client laptop running an i3.

Many of the application may use assembly code. These application perform the same regardless of the of optimization applied by GCC. Common applications include openssl, various video codec libraries, prime95 etc... but I'm not entirely sure how much of assembly they're using; this is the reason why I chose sparsely used algos in openssl for benchmark purposes since the developer is less likely to do efforts for a less used algo.

Many applications are not bottlenecked by the CPU, even though it may seem so, that's because they put more stress on the memory speeds than the CPU. Even when the memory is the bottleneck, the CPU utilization is reported as 100% because of how closely the memory and CPU work. e.g. is compression workloads. In these benchmarks, there will not be much of a difference.

For the source of the benchmark download from here. These are it's contents -- 

script.sh -- The script which was run for the benchmark

shell-bench.sh -- Grep and bash benchmark script.

All outputs of scripts.

Integrating sccache-dist in portage/gentoo for rust distributed compiling.

 NOTE: it did not work out. The (remote) sccache-dist server error out with -- 

Missing output path "/tmp/portage/www-client/firefox-128.12.0/work/firefox_build/instrumented/x86_64-unknown-linux-gnu/release/deps/fallible_iterator-9ab9b312481cd614.d"

The build.log you'll get -- 

Could not perform distributed compile, falling back to local: failed to rewrite outputs from compile: No outputs matched dep info file /tmp/portage/www-client/firefox-128.12.0/work/firefox_build/instrumented/release/deps/unicode_ident-9a905afffc6beb9c.d

And the compile happen locally, not at the remote sccache-dist server. 

The main difference between my approach and this article is that I rely on RUSTC_WRAPPER (and cargo) to pass on the compilation process to the remote build server where as this article believes in creating symlink system after which sccache must work for all packages, not only just rust which was something I was not trying to achieve.

Also the toolchain will be copied over from the client (the machine which is actually initiating the compiling) to the build server. In case the toolchain's binaries are not compatible with the build server (which IS probably the case with most of gentoo's toolchain, but not with rust-bin), it won't workout. The toolchain binaries will error out on the build server.

Regardless, let's begin.

First sccache must be build with dist-client dist-server on both the client and build server.

Add the following to make.conf of the client machine -- 

RUSTC_WRAPPER=/usr/bin/sccache
SCCACHE_MAX_FRAME_LENGTH=104857600
SCCACHE_IGNORE_SERVER_IO_ERROR=0
SCCACHE_DIR=/var/tmp/sccache
SCCACHE_CACHE_SIZE=5G
SCCACHE_CONF=/etc/sccache-client.toml
FEATURES="-ipc-sandbox -network-sandbox -network-sandbox-proxy -pid-sandbox"

Create the client config on the client machine -- 

[dist]
scheduler_url = "http://<IP address of you build server>:64888"
toolchains = []
cache_dir = "/var/tmp/sccache-dist"
[dist.auth]
type = "token"
token = "<a plane text secret>"

And save it as /etc/sccache-client.toml

chown portage:portage /var/tmp/sccache-dist /var/tmp/sccache

Create scheduler config on the builder machine -- 

public_addr = "0.0.0.0:64888"
[client_auth]
type = "token"
token = "<a plane text secret>"
[server_auth]
type = "jwt_hs256"
secret_key = "<generate using `sccache-dist auth generate-shared-token`"

Assuming file is saved as /etc/sccache-sched.toml.

Run the scheduler (as user) --

SCCACHE_NO_DAEMON=1 sccache-dist scheduler --config /etc/sccache-sched.toml

Create build server config --

public_addr = "<IP address of you build server>:8889"
scheduler_url = "http://<IP address of you build server>:64888"
cache_dir = "/var/tmp/sccache_server_toolchain/"
[builder]
type = "overlay"
build_dir = "/var/tmp/sccache_builddir/"
bwrap_path = "/usr/bin/bwrap"
[scheduler_auth]
type = "jwt_token"
token = "<generate using `sccache-dist auth generate-jwt-hs256-server-token --server <IP address of you build server>:8889 --config /etc/sccache-sched.toml `"

Assume config location as /etc/sccache-build.toml.

Run the build server (as root) -- 

SCCACHE_NO_DAEMON=1 sccache-dist server --config /etc/sccache-build.toml