Skip to content

Continuous delivery of the images (design, not implemented yet)

The images follow upstream Exherbo as closely as possible: every update of the package repositories goes through an automated build and tests, becomes a release, and reaches the clusters through release channels, without manual work in the normal case. Exherbo upgrades do break sometimes (resolution blockers, build failures, behaviour changes): they are caught in the pipeline, on a small update, before any cluster gets them.

Ground rule (CLAUDE.md): everything the automation does, a human admin can do the same way. Every step below is a command an admin can run from a workstation (ansible-playbook, etherectl), with the same inputs and the same result; the pipeline and the operator only call these commands.

The chain

  1. Pinned inputs: the Exherbo repositories are git submodules of paludis/repositories/; a commit of ethere (plus the commit of the cluster's configuration) fixes the exact package versions. A scheduled job moves the submodules forward and opens a change (like Renovate, for Exherbo).
  2. Build and tests of the change, with what exists today: builder update (builder/setup.yml, mostly from the builder's pbins), images (images.yml, from their pbins: only updated packages are rebuilt, everything once when the toolchain fingerprint changes), offline inspection, QEMU boots, then the test cloud (Nova, anycast, etcd cluster, later Kubernetes).
  3. Release: images versioned (commit, toolchain fingerprint) uploaded to the Glance of every target cloud, with credentials scoped to that (derived by the credentials layer, secrets.md), never replacing the previous versions; plus a release manifest (image and version per role), signed (e.g. cosign).
  4. Channels: directories or branches of a releases git repository:

    • dev: every release that passes the pipeline;
    • beta: promoted after N days on dev without incident;
    • stable: promoted after N days on beta, automatically on health criteria or with a manual approval.

    A promotion is a commit in that repository: a human can do it too.

  5. Clusters subscribe to a channel (e.g. the test cluster to dev, production to stable): a GitOps controller syncs the release manifest into the cluster, and a new image becomes a new machine template; machines are replaced through Cluster API (cluster-api.md), ethere's provider doing the DNS/TTL choreography of rotations.

Safeguards (the cluster updates its own nodes)

  • etcd and control plane nodes are replaced one at a time, with health checks between steps: etcd quorum and member list, API reachable, nodes Ready, services answering through their DNS names.
  • A failed check stops the rollout; the previous images stay in Glance, so going back is pinning the previous release in git.
  • A cluster can be frozen on a release (pinned in git) during an incident or a critical period.
  • The same procedure — replace the instances of a role one by one with a given image, checking health between steps — exists as a command an admin runs by hand.

GitOps controller: Flux

Flux (a set of controllers driven by CRDs, no UI, built for automation, including controllers that update git when new versions appear) rather than Argo CD (rich web UI, app-of-apps, heavier). Not urgent: both fit.

The pipeline runs on the cluster

  • Trigger: Flux watches the sources (Exherbo repositories, ethere) and applies a pipeline run of a Kubernetes-native CI engine (Tekton PipelineRun or Argo Workflows): Flux doesn't run jobs itself.
  • Builder node: a dedicated bare metal node (e.g. OVHcloud Public Cloud Metal Instances, provisioned by Nova and billed hourly): native KVM for the QEMU boots and the Kolla test cloud (no nested virtualization), many cores, local NVMe for the builder and the pbins cache (mirrored to S3 so that a new node starts warm).
  • On demand: the pipeline requests a builder node and the operator creates it, then deletes it following the hourly billing — the same mechanism as the on-demand GPU nodes (gpu.md).
  • Isolation: the builder node holds what publishes images and signs releases, i.e. what could compromise every cluster: dedicated node (taint), namespace, NetworkPolicy, credentials scoped to publishing.
  • Rootless build in a pod: etherectl ns relies on user namespaces (subordinate ids); in a pod it needs Kubernetes user namespaces (hostUsers: false) or a privileged pod on the isolated builder node. To validate.
  • Chicken and egg: if a release breaks the cluster, its pipeline goes down with it. Hence the channels (the cluster running the pipeline follows stable, new releases reach dev/beta elsewhere first) and the ground rule: the whole chain still runs from a workstation, as it does today.

Prerequisites

Kubernetes on top of etcd, the operator and ethere's Cluster API providers (cluster-api.md), on-demand nodes (gpu.md), versioned images in Glance (today images/_finalize.yml deletes and recreates the image of the same name), release signing, scoped publishing credentials (secrets.md).