Continuous delivery of the images (design, not implemented yet)¶
The images follow upstream Exherbo as closely as possible: every update of the package repositories goes through an automated build and tests, becomes a release, and reaches the clusters through release channels, without manual work in the normal case. Exherbo upgrades do break sometimes (resolution blockers, build failures, behaviour changes): they are caught in the pipeline, on a small update, before any cluster gets them.
Ground rule (CLAUDE.md): everything the automation does, a human admin can do the same way. Every step below
is a command an admin can run from a workstation (ansible-playbook, etherectl), with the same inputs and the
same result; the pipeline and the operator only call these commands.
The chain¶
- Pinned inputs: the Exherbo repositories are git submodules of
paludis/repositories/; a commit of ethere (plus the commit of the cluster's configuration) fixes the exact package versions. A scheduled job moves the submodules forward and opens a change (like Renovate, for Exherbo). - Build and tests of the change, with what exists today: builder update (
builder/setup.yml, mostly from the builder's pbins), images (images.yml, from their pbins: only updated packages are rebuilt, everything once when the toolchain fingerprint changes), offline inspection, QEMU boots, then the test cloud (Nova, anycast, etcd cluster, later Kubernetes). - Release: images versioned (commit, toolchain fingerprint) uploaded to the Glance of every target cloud,
with credentials scoped to that (derived by the credentials layer,
secrets.md), never replacing the previous versions; plus a release manifest (image and version per role), signed (e.g. cosign). -
Channels: directories or branches of a releases git repository:
dev: every release that passes the pipeline;beta: promoted after N days ondevwithout incident;stable: promoted after N days onbeta, automatically on health criteria or with a manual approval.
A promotion is a commit in that repository: a human can do it too.
-
Clusters subscribe to a channel (e.g. the test cluster to
dev, production tostable): a GitOps controller syncs the release manifest into the cluster, and a new image becomes a new machine template; machines are replaced through Cluster API (cluster-api.md), ethere's provider doing the DNS/TTL choreography of rotations.
Safeguards (the cluster updates its own nodes)¶
- etcd and control plane nodes are replaced one at a time, with health checks between steps: etcd quorum
and member list, API reachable, nodes
Ready, services answering through their DNS names. - A failed check stops the rollout; the previous images stay in Glance, so going back is pinning the previous release in git.
- A cluster can be frozen on a release (pinned in git) during an incident or a critical period.
- The same procedure — replace the instances of a role one by one with a given image, checking health between steps — exists as a command an admin runs by hand.
GitOps controller: Flux¶
Flux (a set of controllers driven by CRDs, no UI, built for automation, including controllers that update git when new versions appear) rather than Argo CD (rich web UI, app-of-apps, heavier). Not urgent: both fit.
The pipeline runs on the cluster¶
- Trigger: Flux watches the sources (Exherbo repositories, ethere) and applies a pipeline run of a
Kubernetes-native CI engine (Tekton
PipelineRunor Argo Workflows): Flux doesn't run jobs itself. - Builder node: a dedicated bare metal node (e.g. OVHcloud Public Cloud Metal Instances, provisioned by Nova and billed hourly): native KVM for the QEMU boots and the Kolla test cloud (no nested virtualization), many cores, local NVMe for the builder and the pbins cache (mirrored to S3 so that a new node starts warm).
- On demand: the pipeline requests a
buildernode and the operator creates it, then deletes it following the hourly billing — the same mechanism as the on-demand GPU nodes (gpu.md). - Isolation: the builder node holds what publishes images and signs releases, i.e. what could compromise every cluster: dedicated node (taint), namespace, NetworkPolicy, credentials scoped to publishing.
- Rootless build in a pod:
etherectl nsrelies on user namespaces (subordinate ids); in a pod it needs Kubernetes user namespaces (hostUsers: false) or a privileged pod on the isolated builder node. To validate. - Chicken and egg: if a release breaks the cluster, its pipeline goes down with it. Hence the channels (the
cluster running the pipeline follows
stable, new releases reachdev/betaelsewhere first) and the ground rule: the whole chain still runs from a workstation, as it does today.
Prerequisites¶
Kubernetes on top of etcd, the operator and ethere's Cluster API providers (cluster-api.md), on-demand nodes
(gpu.md), versioned images in Glance (today images/_finalize.yml deletes and recreates the image of the same
name), release signing, scoped publishing credentials (secrets.md).