Nix Fleet, Part 2: Remote Builders — Never Compile Linux on a Mac Again

2026-06-26

Parker Jones and Claude Opus 4.8 in homelab

#nix , #nixos , #remote-builders , #macos , #binfmt , #cross-compilation and #infrastructure

5 minute read

Nix Fleet, part 2. Part 1 laid out the one-flake-every-machine architecture. This part solves the problem that makes a mixed Linux/macOS fleet painful: my laptop can't build Linux.

When you develop NixOS configs on a Mac, you hit a wall fast: an ARM macOS machine cannot build x86_64-linux or aarch64-linux derivations locally. Nothing about Apple Silicon produces Linux binaries. So either you accept that you can only iterate on Linux configs from a Linux machine, or you set up a remote builder and make the problem disappear.

I set up the remote builder. Now my M3 laptop offloads every Linux build to a beefy x86_64 NixOS box, and from where I sit it's transparent — nix build just works, the fans on the laptop stay quiet, and the heavy lifting happens on a machine designed for it.

The shape of the solution

There are two roles:

The builder — my always-on x86_64 NixOS box (SSH alias nixos). It has the cores, the RAM, and the right architecture.
The controller — my M3 MacBook. It evaluates the flake and orchestrates, but delegates the actual compilation.

The trick that makes this clean: configure the controller so that it does zero local jobs and ships everything to the builder.

Builder side: accept work, and emulate ARM

The builder needs to be reachable over SSH and willing to build for the architectures I care about. The one non-obvious piece is ARM: I also run an aarch64-linux machine (a Helios64 NAS), and rather than stand up a separate ARM builder, I let the x86 box build ARM derivations through binfmt/QEMU emulation:

# on the x86_64 builder
boot.binfmt.emulatedSystems = [ "aarch64-linux" ];

That one line registers a QEMU user-mode handler so the x86 host transparently runs aarch64 build steps. It's slower than native ARM (more on that below), but it means one builder covers my entire Linux fleet across both architectures.

Controller side: offload everything

On the Mac, I point Nix at the builder and tell it not to build anything itself:

# /etc/nix/nix.conf (or the equivalent NixOS/darwin option)
builders = @/etc/nix/machines
builders-use-substitutes = true
max-jobs = 0

# /etc/nix/machines
nixos x86_64-linux / - 8 1 kvm,big-parallel

The magic value is max-jobs = 0. It tells the local machine to run zero build jobs — so every derivation that isn't already in a binary cache must go to a remote builder. There's no "sometimes it builds locally and melts the laptop" ambiguity; local builds are simply off. builders-use-substitutes = true lets the builder pull from binary caches directly instead of having the controller ferry everything, which saves a lot of pointless data movement.

The /etc/nix/machines line reads as: builder nixos, builds x86_64-linux, default SSH key, up to 8 parallel jobs, speed factor 1, and it supports the kvm and big-parallel features (so it'll accept jobs that need KVM or heavy parallelism).

Verifying it actually offloads

The failure mode here is silent: you think you're offloading and you're not. So verify. First, list what the flake even knows how to build:

nix eval .#nixosConfigurations --apply builtins.attrNames

Then kick off a real Linux build from the Mac and watch the builder, not the laptop:

nix build .#nixosConfigurations.<host>.config.system.build.toplevel

If it's working, the laptop sits nearly idle while the x86 box lights up. If max-jobs is misconfigured, you'll feel it — the Mac will either try (and fail, for Linux) or grind. Watching nproc/free -h/build load on the builder during a build is the quickest confirmation.

When builds go wrong

Offloading introduces its own failure modes, and I keep a short runbook for them:

OOM on the builder, especially during ARM-emulated builds, which are memory-hungry. Mitigations: serialize with --option max-jobs 1 --option cores 2, or add temporary zram swap:
```
{ zramSwap = { enable = true; memoryPercent = 50; }; }
```
Inspecting a failed derivation without re-running the whole build:
```
nix log /nix/store/<hash>-<drv>.drv | less -R
```

Checking effective settings when something isn't offloading as expected:

nix show-config | egrep '^(cores|max-jobs|builders|system-features|substituters)'

The honest trade-off

Emulated ARM builds are slow — QEMU user-mode emulation has real overhead, and a big aarch64 derivation built on x86 can take noticeably longer than it would native. For my fleet that's an acceptable price: I build ARM rarely (the NAS config doesn't change often), and the alternative is maintaining a second physical ARM builder. If you're iterating heavily on ARM, buy or borrow a native aarch64 builder and add it to /etc/nix/machines as a second entry — the controller config doesn't otherwise change. The architecture scales to N builders the same way it handles one.

The win is the same one Part 1 was about: I get to live entirely in macOS, drive my whole Linux fleet from there, and never once wait on my laptop to compile something it was never meant to compile. Part 3 tackles the next fleet-wide concern — secrets — including how a first-boot install fetches them securely over the same SSH trust I just set up here.

The builder/controller config is part of my parallaxisjones/dotfiles repo.

— Parker Jones, parkerjones.dev