Skip to content

Cluster API providers

ethere relies on the core of Cluster API (CAPI) and implements its own providers, with Ansible only (no cluster-api-provider-openstack, no OpenTofu): the operator's playbooks (docs/admin/operator.md).

Where it stands

Done (contract v1beta2, CAPI v1.14.3, tests/cluster/capi.test.js):

  • The core as an addon of the control plane (inventory/group_vars/k8s/cluster-api.yml): core-components.yaml as published, pinned by sha256, rendered at the image's build (its variables' defaults; capi-system out of the default policies: the apiserver reaches its webhooks), its controller's image embedded; its certificates from cert-manager. The CRDs of ethere's kinds with it, labelled cluster.x-k8s.io/v1beta2: v1alpha1 (a reference carries no version: CAPI reads it there), in the groups infrastructure.cluster.x-k8s.io and bootstrap.cluster.x-k8s.io, on which CAPI's core already has every right. A control plane with the operator and CAPI wants 4 vCPU and 8 GiB: on 2 vCPU and 3 GiB the controllers lose their leases and restart in turn.
  • EthereCluster (playbooks/capi/cluster.yml): nothing to make on the cloud: spec.controlPlaneEndpoint (the API's public name through the edges), status.initialization.provisioned, the zones as status.failureDomains, and the Secret <cluster>-kubeconfig CAPI reaches the cluster with (the cluster itself: the kubernetes Service, a client certificate of capi_kubeconfig_hours issued again with a third left).
  • EthereConfig, EthereConfigTemplate (playbooks/capi/config.yml): spec.image; the bootstrap data is the image's user-data (provisioning/_userdata.yml, shared with provisioning/servers.yml: nothing in it is specific to a server), in a Secret of its name it owns.
  • EthereMachine, EthereMachineTemplate (playbooks/capi/machine.yml): spec.image, spec.flavor; the server has the resource's name, created with the Machine's bootstrap data (provisioning/_server.yml) in its failure domain or the least used zone, its cluster mode the annotation ethe.re/cluster-mode (default join), enrolled through its console; a run interrupted resumes (the server found by its name). spec.providerID is <provider>:///<UUID>, which the kubelet registers with (ethere-netconfig): CAPI finds the Node by it. The annotation ethe.re/adopt: <server> reports an existing server instead. Its finalizer deletes the server and its ports (provisioning/delete.yml).
  • EthereControlPlane (playbooks/capi/controlplane.yml, _hooks.yml, _machine_add.yml, its scale subresource): spec.replicas, spec.version, spec.machineTemplate.spec.infrastructureRef (an EthereMachineTemplate), spec.rollout.after. Stateless, one step per run, every minute, one at a time: an operator evicted in the middle (it runs on these servers: it was, by the drain, in the first rollout tested) goes on where the Machines are. Each run writes its status (the counters from the Machines' conditions, selector, versions, initialization.controlPlaneInitialized), then the first that applies:

    1. a Machine without a controller owner claimed (the owner, the label cluster.x-k8s.io/control-plane-name, the two hooks): capi/init.yml's;
    2. a Machine being deleted: the hook CAPI waits for (its condition Deleting's reason), one Machine at a time, nothing else meanwhile. provisioning/replace.yml's steps, around what CAPI does itself (the drain, the volumes, the Node's deletion). pre-drain.delete.hook.machine.cluster.x-k8s.io/ethere: the git writer moved if it's this one, to the oldest other replica of the current template (once in a rollout), else the oldest other one, the edges' name without it (k8s/gateway.yml) and the resolvers' caches; on qemu, the host's ports to another server. pre-terminate.delete.hook.machine.cluster.x-k8s.io/ethere: the hidden primary elsewhere serving the DNSEndpoints' records, then the server stopped and its etcd member removed, in the order the quorum allows (two members left: the member first, while it runs; a server already down: the member alone). The finalizer of its EthereMachine then deletes the server;
    3. an unhealthy Machine (OwnerRemediated False: a MachineHealthCheck on the label cluster.x-k8s.io/control-plane) deleted if the others hold the quorum: a dead member goes first (a fourth member couldn't join), then the count is short. A lost quorum isn't its: there is no API (etcd/recover.yml, an admin or a management cluster);
    4. a Machine not Ready yet: wait;
    5. fewer than spec.replicas: a Machine added (bootstrap if the cluster isn't initialized, join otherwise; the least used failure domain);
    6. the git names out of date (git/writer.yml): a new replica's; one gone's with the next new one's, or at the rollout's end;
    7. an outdated Machine (its EthereMachine's image or flavor not the template's, its version not spec.version, older than spec.rollout.after or than rotate_due_days: the servers' certificates): surge first, one more added, then, its etcd member answering, the oldest outdated deleted; more than spec.replicas without one: the oldest deleted.

    It rotates the control plane by age itself: operator/reconcile.yml leaves the images of k8s_ctrl alone once an EthereControlPlane exists. A server takes about 25 min on the test cloud, with 4 vCPU (about 8 to join, 5 for its git name and etcd member, 12 for its hooks and deletion).

  • playbooks/capi/init.yml: over a cluster the playbooks made, in the operator's namespace: the Cluster (its controlPlaneRef), its EthereCluster, the EthereControlPlane and its template, and each server of the etcd image adopted as a control plane Machine (label cluster.x-k8s.io/control-plane, a Secret standing for its bootstrap data), which the EthereControlPlane claims. Workers can then be MachineDeployments.

Waits are the watches' periods (operator_watches): a playbook returns when something isn't there yet and runs again. Their objects are read through the admin's rights of the playbooks; status is written with kubectl patch --subresource=status (manageStatus: false), initialization and failureDomains never empty.

Waits are watch periods because the playbooks act with their own admin certificate: ansible-operator's dependent watches only see what goes through its proxy.

On the qemu provider (docs/admin/standalone.md, tests/standalone/standalone.test.js): the operator in the standalone's own VM, asking the host through the metadata service; a rollout with replicas: 1 takes about 20 min at 4 vCPU: a new VM joins (two etcd members), the host's ports move to it, the old one drains, its volumes follow their pods, its member goes while it runs, it stops.

The two modes

  • Internal (a cluster managing itself): capi/init.yml as above, then kubernetes.default.svc for CAPI, the operator on a control plane server's host network, the edge on its host.
  • External (operator_kubeconfig: a management cluster's kubeconfig; tests/cluster/external.test.js): capi/init.yml from the workstation, over a cluster the playbooks made, installs the operator and declares the objects on the management cluster, in a namespace per cluster (ethere-<cluster>), and leaves nothing of the operator's in the managed cluster. A MachineHealthCheck comes with capi/init.yml in both modes, for more than one server. There:
    • operator_kubeconfig is where the CAPI objects and the operator are (in its pod, its ServiceAccount's), the admin kubeconfig the managed cluster (etcd, git, DNS);
    • CAPI's <cluster>-kubeconfig and the operator reach the cluster through api.<zone>, the operator's git through the first of its addresses that answers (edge_by_name: a stopped edge stays in DNS until its replacement takes it out);
    • the operator is on the pods' network by default (operator_host_network: false), its egress a CiliumNetworkPolicy (kube-apiserver, world; playbooks/operator/_network.yml), DNS64 and NAT64 for IPv4 edges;
    • the git bundle must be in S3 (git_bundle_in_cluster is refused: the Secret would be in the cluster to recover).

Tested on the test cloud: a management cluster of one aio manages three, in two OpenStack projects with a member's rights; the servers adopted, the Secret issued again with CAPI still connected, the git writer's server stopped and replaced. Two things it does not test: the operator reaching OpenStack from the pods' network (the test cloud's catalog gives literal IPv4 addresses, unreachable from IPv6 pods: its operator is on the host's network, and a separate pod checks the policy for the API and the public name), and a real api.<zone> (a fake DNS server answers the name, tests/fixtures/cluster/outside_dns.py).

The git writer's server down, the PKI isn't read (operator/pki.yml goes on with pki_dir as it is, until the next run), so that the hook moving the writer gets its turn.

An annotation on an EthereCluster runs it at once (watchAnnotationsChanges), otherwise every operator_reconcile_period.

A lost quorum or a lost cluster, on request

From a management cluster, an admin's annotation on the EthereCluster (kubectl annotate etherecluster <cluster> ethe.re/recover=<anything>) runs playbooks/capi/recover.yml. It is never automatic: a network partition seen from the management cluster must not delete servers. provisioning/cluster.yml still asks Nova twice before it acts. The run:

  1. pauses Cluster API (the Cluster's spec.paused, the EthereControlPlane's annotation);
  2. runs provisioning/cluster.yml (a recovery with a member left, a restore without any). It refuses to make a new empty cluster (cluster_new, or no backup: an admin's decision) and leaves alone the credentials an operator has no right to make, which etcd keeps;
  3. adopts the new servers (capi/init.yml), deletes the Machines of servers gone without their hooks;
  4. resumes Cluster API and removes the annotation last: a failed run goes on where it was, except for a restore interrupted before its server got the snapshot's URL (TODO.md). The annotation gone means Cluster API resumed: the new Machines still have to become Running.

DNS still gives a lost cluster's old edges until the rebuilt one publishes its own, so the playbooks use the first address of the name that answers, else Nova's addresses (edge_nat64_prefix on the pods' network; the same from a workstation). Only an operator on a management cluster reads the etcd backup: its S3 user's policy has one more statement (operator/credentials.yml). That backup holds every Secret of the cluster; the operator, which holds its CA keys, could already issue itself their access.

A restore goes back to the last backup: what changed since is lost.

The design

What CAPI brings

Its core controllers and ecosystem:

  • Machine, MachineSet, MachineDeployment: progressive replacement of machines when the image changes — the cattle rotation;
  • MachineHealthCheck: automatic replacement of unhealthy nodes;
  • Cluster Autoscaler integration (clusterapi provider, scale from zero): a base for on-demand GPU nodes (gpu.md), the flavor choice and billing-aligned scale-down staying ours (driving replicas);
  • clusterctl, conventions, tooling.

What ethere provides

One Ansible operator (Operator SDK), whose roles are shared with the bootstrap playbooks (ethere Ansible collection):

CAPI contract ethere resources Responsibility
Infrastructure cluster EthereCluster private networks, subnet pool, anycast ports, server groups, API endpoint (anycast / DNS), availability zones as failure domains, provider profile (providers.md)
Infrastructure machine EthereMachine, EthereMachineTemplate server creation (ports per role network, volumes, image, flavor, AZ, server group), providerID, addresses; deletion with ports/volumes cleanup (playbooks/provisioning/delete.yml does it for the floating IPs and ports; no volumes today)
Bootstrap EthereConfig the minimal user-data of ethere images (zero post-boot configuration): the same for every server of an image, the network read from the metadata service (sys-apps/ethere-netconfig), the cluster mode from the server's metadata, the kubelet's providerID openstack:///<UUID>, the sentinel /run/cluster-api/bootstrap-success.complete
Control plane EthereControlPlane all-in-one or dedicated roles, etcd membership (join through anycast, removal of dead members), rotations
  • DNS choreography: rollouts don't wait for DNS TTLs by themselves; the DNS controller holds machine deletion through CAPI lifecycle hooks (pre-drain / pre-terminate annotations on Machines, Runtime SDK) until the record change has propagated (dns.md).
  • Management cluster: CAPI needs one; the cluster manages itself (self-hosted). Two ways to get there, both driven by the ethere bootstrap (Ansible):
    • Adopt, then rotate (preferred, to evaluate first in the spike): the bootstrap playbooks create the first cluster as today (possibly minimal), install CAPI and the ethere providers in it, then declare the existing state: an EthereCluster marked as externally managed (cluster.x-k8s.io/managed-by annotation: CAPI reads it, creates nothing), and Machine/EthereMachine objects bound to the existing servers by providerID (the infrastructure provider only needs an "adopt" path: find the server, report it ready, create nothing). A rollout then replaces the adopted machines one by one with CAPI-native ones: adoption only needs to be good enough to be replaced, and after the first rotation the cluster is fully native. No temporary cluster, Ansible end to end, and the same path takes over an existing cluster or finishes a disaster recovery rebuilt by the playbooks. To validate first: how CAPI lets MachineSets/MachineDeployments take adopted (ownerless) Machines over through their labels; the control plane provider must accept existing etcd members and control plane nodes.
    • Temporary cluster (standard CAPI path): start a single-node k0s (k0s controller --single: one static binary embedding containerd, no Docker/podman needed, unlike kind) in a throwaway instance on the target cloud or in a rootless local QEMU VM; clusterctl init with CAPI and the ethere providers; apply the cluster manifests (the providers create the cluster); clusterctl move to it; destroy the temporary cluster.
  • Provider images: the ethere providers (Ansible operator) are container images, to be published in a registry reachable by both the temporary and the target clusters.
  • Why Ansible only: one implementation for bootstrap and steady state; idempotent openstack.cloud modules; the Ansible operator model (a role per reconciliation, finalizers, k8s_status) fits the provider contracts. OpenTofu is good for static declarative infrastructure but ill-suited to per-machine reconciliation in a controller (state and lock per machine, slow plan/apply on each event), and a second tool doubles maintenance. CAPO is not used because its model doesn't fit ethere's (anycast ports, role networks, Octavia by default, user-data).

Risks

  • The infrastructure provider contract is demanding (status conditions, immutable templates, failure domains, deletion semantics); CAPO matured on many edge cases we will meet too.
  • The contract evolves (v1beta1 → v1beta2): follow CAPI releases.
  • CAPI's own end-to-end test framework is in Go; ethere tests through its Bun suite on the test cloud.