Skip to content

Host setup

What a workstation needs to run ethere (image builds, provisioning, tests). Written for Ubuntu 24.04 (26.04: see the packages below); other distributions need the same tools and the same user namespace permissions.

Everything runs as a regular user (see "Environment" in CLAUDE.md): sudo is only needed once, for the steps below.

Packages

sudo apt install git curl rsync jq unzip xz-utils zstd cpio make gcc \
  uidmap golang-go erofs-utils dosfstools mtools qemu-utils qemu-system-x86 ovmf yamllint
sudo apt install systemd-repart   # Ubuntu 24.10 and later: no longer part of the systemd package

What needs them:

  • unzip: the asdf plugins of bun and awscli unpack zip archives.
  • uidmap (newuidmap/newgidmap): etherectl ns, i.e. unshare --map-auto (image builds, etherectl chroot).
  • golang-go: the profile installs crane, helm and regctl with go install.
  • erofs-utils, dosfstools, mtools: assembling the disk images with systemd-repart (part of the systemd package on Ubuntu 24.04, its own package since 24.10), without mounts: erofs root, vfat ESP filled with mcopy.
  • qemu-utils (qemu-img): raw → qcow2 conversion. qemu-system-x86, ovmf (UEFI firmware): the QEMU boot tests.
  • yamllint: yamllint . (linting the YAML).

Everything else (asdf, uv and a uv-managed Python, bun, sops, age, awscli, Ansible and its collections) is installed by the profile under $ETHERE_CLUSTER_ROOT_DIR/.local/ on the first etherectl shell.

Subordinate ids

etherectl ns maps uid/gid 1..65536 inside the namespace to the user's subordinate ids. Ubuntu creates them with the user; check:

grep "^$USER:" /etc/subuid /etc/subgid   # e.g. romain:100000:65536

If missing: sudo usermod --add-subuids 100000-165535 --add-subgids 100000-165535 $USER.

AppArmor: unprivileged user namespaces

Ubuntu 24.04 restricts unprivileged user namespaces (kernel.apparmor_restrict_unprivileged_userns = 1): a namespace created by an unconfined program is moved to the unprivileged_userns profile, which denies mounts and capabilities. etherectl ns then fails with:

unshare: cannot change root filesystem propagation: Permission denied

Allow it for unshare only, with a profile (the restriction stays for everything else):

printf 'abi <abi/4.0>,\ninclude <tunables/global>\n\nprofile unshare /usr/bin/unshare flags=(unconfined) {\n  userns,\n  include if exists <local/unshare>\n}\n' \
  | sudo tee /etc/apparmor.d/unshare-userns
sudo apparmor_parser -r /etc/apparmor.d/unshare-userns

The profile is loaded again at every boot. Writing the file is not enough: apparmor_parser -r loads it now. Disabling the restriction globally (sysctl kernel.apparmor_restrict_unprivileged_userns=0) works too, but is not recommended.

Check:

unshare --map-auto --map-root-user --mount --pid --fork --kill-child --propagation private \
  sh -c 'id; cat /proc/self/uid_map'
# uid=0(root) ...
#          0       1000          1
#          1     100000      65536

If it still fails, journalctl -k | grep apparmor shows which profile denied what.

KVM

The QEMU boot tests and the test cloud (etherectl test-cloud, a Kolla AIO VM) run QEMU with accel=kvm: the user must be able to open /dev/kvm, which belongs to the kvm group on Ubuntu.

sudo usermod -aG kvm $USER   # then log out and back in
python3 -c "import os; os.close(os.open('/dev/kvm', os.O_RDWR)); print('kvm ok')"

Memory for the builds (optional)

With enough RAM, paludis's build directory can live in memory: builder_tmpfs_size: 24G in the cluster's inventory/group_vars/all/ mounts a tmpfs of that size on the builder's /var/tmp/paludis/build for each run (builder/mounts.yml). The compilations' temporary files (gigabytes per large package, deleted right after) then never reach the disk. 24G covers the largest packages (llvm, rust); the memory is only used while something compiles.

LXD (optional): the test cloud without nested virtualization

The test cloud's default backend is a QEMU VM: its instances are nested VMs. On a CPU without VMCS shadowing nor APICv (Intel before Haswell, e.g. Sandy Bridge: cat /sys/module/kvm_intel/parameters/enable_shadow_vmcs prints N), every exit of a nested VM goes through the host, and they boot 2 to 3 times slower. The lxd backend runs Kolla in a privileged LXD system container instead, sharing the host's kernel and /dev/kvm: the instances are plain VMs of the host. Measured 2026-10-25 on a Xeon E5-2430L, an ethere-quorum instance (2 vCPU, 2 GiB):

QEMU on the host qemu backend lxd backend
multi-user.target (guest kernel time) 9 s 21 s 11 s
report on the console, after Nova's ACTIVE — 38 s 12–16 s

Nova itself still takes ~22 s to make an instance ACTIVE. The cluster tests gain less, being mostly Ansible's time: tests/cluster/pki.test.js took 31 min instead of 36. The qemu VM also used two of the host's threads at idle; the container, Kolla's services only.

sudo snap install lxd                 # Ubuntu 26.04: the lxd-installer wrapper does it on the first `lxc` call
sudo usermod -aG lxd $USER            # then log out and back in
lxd init --auto --storage-backend=btrfs --storage-create-loop=60
etherectl test-cloud up --backend lxd   # first use only; `destroy` first to change the backend
  • The lxd group is root-equivalent (it can start privileged containers and mount the host's filesystems): the one exception to "no root" for this workstation, and only for the test cloud. etherectl test-cloud never calls sudo; LXD's daemon loads the kernel modules the container needs (linux.kernel_modules).
  • lxd init --auto creates the bridge lxdbr0 (a /24 with NAT; the container gets its .10, which the host reaches directly: no proxy) and the storage pool default. Its storage must snapshot cheaply (btrfs or zfs: the installed state is a snapshot): on an ext4 root, a btrfs pool on a loop file, sized for Kolla's images and the instances (60 GB, sparse). On a btrfs root, --storage-backend=btrfs without a loop uses the filesystem itself.
  • The container is privileged (security.privileged, security.nesting, no AppArmor confinement, every device and capability: Docker, Open vSwitch, libvirt): it can change the host's global sysctls and network state. Kolla's bootstrap-servers sets a few. Fine for a workstation dedicated to it; think twice elsewhere.

Moving to another machine

A cluster's configuration only needs ethere's main clone next to it (git clone it to ~/ethere, link bin/etherectl into the PATH): playbooks that don't build use the images already in Glance. The build workspace ($ETHERE_BUILD_DIR, the main clone) moves only to build there. Don't copy its builder/ and images/ with a plain rsync -a: they hold files owned by the subordinate ids, which a regular user can neither read (permission errors) nor recreate (everything ends up owned by the user). Either rebuild them (downloads/ and pbins/ can be copied as is):

etherectl ns rm -rf "$ETHERE_BUILD_DIR/builder" "$ETHERE_BUILD_DIR/images"
etherectl ns ansible-playbook $ETHERE_ROOT_DIR/playbooks/builder/setup.yml
etherectl ns ansible-playbook $ETHERE_ROOT_DIR/playbooks/images.yml

or copy them through the namespace on both sides, which keeps the numeric owners (etherectl ns tar --numeric-owner -C ~/ethere -cf - builder images on the old machine, piped into etherectl ns tar --numeric-owner -C ~/ethere -xf - on the new one).

In each cluster's configuration, also copy the gitignored files that can't be regenerated: key.enc (the personal age key), and the PKI repo (pki/) if it has no remote.

The test cloud's state (.local/share/ethere/test-cloud/, its provisioned disks) can be copied as is: its disks reference each other by relative paths. Disks created by an older ethere hold absolute paths; see the changelog entry about the test cloud surviving a move to fix them.