Host setup¶
What a workstation needs to run ethere (image builds, provisioning, tests). Written for Ubuntu 24.04 (26.04: see the packages below); other distributions need the same tools and the same user namespace permissions.
Everything runs as a regular user (see "Environment" in CLAUDE.md): sudo is only needed once, for the
steps below.
Packages¶
sudo apt install git curl rsync jq unzip xz-utils zstd cpio make gcc \
uidmap golang-go erofs-utils dosfstools mtools qemu-utils qemu-system-x86 ovmf yamllint
sudo apt install systemd-repart # Ubuntu 24.10 and later: no longer part of the systemd package
What needs them:
unzip: the asdf plugins of bun and awscli unpack zip archives.uidmap(newuidmap/newgidmap):etherectl ns, i.e.unshare --map-auto(image builds,etherectl chroot).golang-go: the profile installs crane, helm and regctl withgo install.erofs-utils,dosfstools,mtools: assembling the disk images withsystemd-repart(part of thesystemdpackage on Ubuntu 24.04, its own package since 24.10), without mounts: erofs root, vfat ESP filled withmcopy.qemu-utils(qemu-img): raw → qcow2 conversion.qemu-system-x86,ovmf(UEFI firmware): the QEMU boot tests.yamllint:yamllint .(linting the YAML).
Everything else (asdf, uv and a uv-managed Python, bun, sops, age, awscli, Ansible and its collections) is installed
by the profile under $ETHERE_CLUSTER_ROOT_DIR/.local/ on the first etherectl shell.
Subordinate ids¶
etherectl ns maps uid/gid 1..65536 inside the namespace to the user's subordinate ids. Ubuntu creates them with
the user; check:
If missing: sudo usermod --add-subuids 100000-165535 --add-subgids 100000-165535 $USER.
AppArmor: unprivileged user namespaces¶
Ubuntu 24.04 restricts unprivileged user namespaces (kernel.apparmor_restrict_unprivileged_userns = 1): a
namespace created by an unconfined program is moved to the unprivileged_userns profile, which denies mounts and
capabilities. etherectl ns then fails with:
Allow it for unshare only, with a profile (the restriction stays for everything else):
printf 'abi <abi/4.0>,\ninclude <tunables/global>\n\nprofile unshare /usr/bin/unshare flags=(unconfined) {\n userns,\n include if exists <local/unshare>\n}\n' \
| sudo tee /etc/apparmor.d/unshare-userns
sudo apparmor_parser -r /etc/apparmor.d/unshare-userns
The profile is loaded again at every boot. Writing the file is not enough: apparmor_parser -r loads it now.
Disabling the restriction globally (sysctl kernel.apparmor_restrict_unprivileged_userns=0) works too, but is not
recommended.
Check:
unshare --map-auto --map-root-user --mount --pid --fork --kill-child --propagation private \
sh -c 'id; cat /proc/self/uid_map'
# uid=0(root) ...
# 0 1000 1
# 1 100000 65536
If it still fails, journalctl -k | grep apparmor shows which profile denied what.
KVM¶
The QEMU boot tests and the test cloud (etherectl test-cloud, a Kolla AIO VM) run QEMU with accel=kvm: the
user must be able to open /dev/kvm, which belongs to the kvm group on Ubuntu.
sudo usermod -aG kvm $USER # then log out and back in
python3 -c "import os; os.close(os.open('/dev/kvm', os.O_RDWR)); print('kvm ok')"
Memory for the builds (optional)¶
With enough RAM, paludis's build directory can live in memory: builder_tmpfs_size: 24G in the cluster's
inventory/group_vars/all/ mounts a tmpfs of that size on the builder's /var/tmp/paludis/build for each run
(builder/mounts.yml). The compilations' temporary files (gigabytes per large package, deleted right after) then
never reach the disk. 24G covers the largest packages (llvm, rust); the memory is only used while something
compiles.
LXD (optional): the test cloud without nested virtualization¶
The test cloud's default backend is a QEMU VM: its instances are nested VMs. On a CPU without VMCS shadowing nor
APICv (Intel before Haswell, e.g. Sandy Bridge: cat /sys/module/kvm_intel/parameters/enable_shadow_vmcs prints
N), every exit of a nested VM goes through the host, and they boot 2 to 3 times slower. The lxd backend runs
Kolla in a privileged LXD system container instead, sharing the host's kernel and /dev/kvm: the instances are plain
VMs of the host. Measured 2026-10-25 on a Xeon E5-2430L, an ethere-quorum instance (2 vCPU, 2 GiB):
| QEMU on the host | qemu backend |
lxd backend |
|
|---|---|---|---|
multi-user.target (guest kernel time) |
9 s | 21 s | 11 s |
| report on the console, after Nova's ACTIVE | — | 38 s | 12–16 s |
Nova itself still takes ~22 s to make an instance ACTIVE. The cluster tests gain less, being mostly Ansible's time:
tests/cluster/pki.test.js took 31 min instead of 36. The qemu VM also used two of the host's threads at idle;
the container, Kolla's services only.
sudo snap install lxd # Ubuntu 26.04: the lxd-installer wrapper does it on the first `lxc` call
sudo usermod -aG lxd $USER # then log out and back in
lxd init --auto --storage-backend=btrfs --storage-create-loop=60
etherectl test-cloud up --backend lxd # first use only; `destroy` first to change the backend
- The
lxdgroup is root-equivalent (it can start privileged containers and mount the host's filesystems): the one exception to "no root" for this workstation, and only for the test cloud.etherectl test-cloudnever calls sudo; LXD's daemon loads the kernel modules the container needs (linux.kernel_modules). lxd init --autocreates the bridgelxdbr0(a /24 with NAT; the container gets its.10, which the host reaches directly: no proxy) and the storage pooldefault. Its storage must snapshot cheaply (btrfs or zfs: the installed state is a snapshot): on an ext4 root, a btrfs pool on a loop file, sized for Kolla's images and the instances (60 GB, sparse). On a btrfs root,--storage-backend=btrfswithout a loop uses the filesystem itself.- The container is privileged (
security.privileged,security.nesting, no AppArmor confinement, every device and capability: Docker, Open vSwitch, libvirt): it can change the host's global sysctls and network state. Kolla'sbootstrap-serverssets a few. Fine for a workstation dedicated to it; think twice elsewhere.
Moving to another machine¶
A cluster's configuration only needs ethere's main clone next to it (git clone it to ~/ethere, link bin/etherectl
into the PATH): playbooks that don't build use the images already in Glance. The build workspace ($ETHERE_BUILD_DIR,
the main clone) moves only to build there. Don't copy its builder/ and images/ with a plain rsync -a: they hold
files owned by the subordinate ids, which a regular user can neither read (permission errors) nor recreate
(everything ends up owned by the user). Either rebuild them (downloads/ and pbins/ can be copied as is):
etherectl ns rm -rf "$ETHERE_BUILD_DIR/builder" "$ETHERE_BUILD_DIR/images"
etherectl ns ansible-playbook $ETHERE_ROOT_DIR/playbooks/builder/setup.yml
etherectl ns ansible-playbook $ETHERE_ROOT_DIR/playbooks/images.yml
or copy them through the namespace on both sides, which keeps the numeric owners (etherectl ns tar --numeric-owner
-C ~/ethere -cf - builder images on the old machine, piped into etherectl ns tar --numeric-owner -C ~/ethere
-xf - on the new one).
In each cluster's configuration, also copy the gitignored files that can't be regenerated: key.enc (the personal age
key), and the PKI repo (pki/) if it has no remote.
The test cloud's state (.local/share/ethere/test-cloud/, its provisioned disks) can be copied as is: its disks
reference each other by relative paths. Disks created by an older ethere hold absolute paths; see the changelog
entry about the test cloud surviving a move to fix them.