Tuesday, August 5, 2025

Integrating sccache-dist in portage/gentoo for rust distributed compiling.

 NOTE: it did not work out. The (remote) sccache-dist server error out with -- 

Missing output path "/tmp/portage/www-client/firefox-128.12.0/work/firefox_build/instrumented/x86_64-unknown-linux-gnu/release/deps/fallible_iterator-9ab9b312481cd614.d"

The build.log you'll get -- 

Could not perform distributed compile, falling back to local: failed to rewrite outputs from compile: No outputs matched dep info file /tmp/portage/www-client/firefox-128.12.0/work/firefox_build/instrumented/release/deps/unicode_ident-9a905afffc6beb9c.d

And the compile happen locally, not at the remote sccache-dist server. 

The main difference between my approach and this article is that I rely on RUSTC_WRAPPER (and cargo) to pass on the compilation process to the remote build server where as this article believes in creating symlink system after which sccache must work for all packages, not only just rust which was something I was not trying to achieve.

Also the toolchain will be copied over from the client (the machine which is actually initiating the compiling) to the build server. In case the toolchain's binaries are not compatible with the build server (which IS probably the case with most of gentoo's toolchain, but not with rust-bin), it won't workout. The toolchain binaries will error out on the build server.

Regardless, let's begin.

First sccache must be build with dist-client dist-server on both the client and build server.

Add the following to make.conf of the client machine -- 

RUSTC_WRAPPER=/usr/bin/sccache
SCCACHE_MAX_FRAME_LENGTH=104857600
SCCACHE_IGNORE_SERVER_IO_ERROR=0
SCCACHE_DIR=/var/tmp/sccache
SCCACHE_CACHE_SIZE=5G
SCCACHE_CONF=/etc/sccache-client.toml
FEATURES="-ipc-sandbox -network-sandbox -network-sandbox-proxy -pid-sandbox"

Create the client config on the client machine -- 

[dist]
scheduler_url = "http://
<IP address of you build server>:64888"
toolchains = []
cache_dir = "/var/tmp/sccache-dist"
[dist.auth]
type = "token"
token = "
<a plane text secret>"

And save it as /etc/sccache-client.toml

chown portage:portage /var/tmp/sccache-dist /var/tmp/sccache

Create scheduler config on the builder machine -- 

public_addr = "0.0.0.0:64888"
[client_auth]
type = "token"
token = "<a plane text secret>"
[server_auth]
type = "jwt_hs256"
secret_key = "<generate using `sccache-dist auth generate-shared-token`"

Assuming file is saved as /etc/sccache-sched.toml.

Run the scheduler (as user) --

SCCACHE_NO_DAEMON=1 sccache-dist scheduler --config /etc/sccache-sched.toml

Create build server config --

public_addr = "<IP address of you build server>:8889"
scheduler_url = "http://
<IP address of you build server>:64888"
cache_dir = "/var/tmp/sccache_server_toolchain/"
[builder]
type = "overlay"
build_dir = "/var/tmp/sccache_builddir/"
bwrap_path = "/usr/bin/bwrap"
[scheduler_auth]
type = "jwt_token"
token = "<generate using `sccache-dist auth generate-jwt-hs256-server-token --server 
<IP address of you build server>:8889 --config /etc/sccache-sched.toml `"

Assume config location as /etc/sccache-build.toml.

Run the build server (as root) -- 

SCCACHE_NO_DAEMON=1 sccache-dist server --config /etc/sccache-build.toml