Cluster API providers¶
ethere relies on the core of Cluster API (CAPI) and implements its own providers, with Ansible only (no cluster-api-provider-openstack, no OpenTofu): the operator's playbooks (docs/admin/operator.md).
Where it stands¶
Done (contract v1beta2, CAPI v1.14.3, tests/cluster/capi.test.js):
- The core as an addon of the control plane (
inventory/group_vars/k8s/cluster-api.yml):core-components.yamlas published, pinned by sha256, rendered at the image's build (its variables' defaults;capi-systemout of the default policies: the apiserver reaches its webhooks), its controller's image embedded; its certificates from cert-manager. The CRDs of ethere's kinds with it, labelledcluster.x-k8s.io/v1beta2: v1alpha1(a reference carries no version: CAPI reads it there), in the groupsinfrastructure.cluster.x-k8s.ioandbootstrap.cluster.x-k8s.io, on which CAPI's core already has every right. A control plane with the operator and CAPI wants 4 vCPU and 8 GiB: on 2 vCPU and 3 GiB the controllers lose their leases and restart in turn. EthereCluster(playbooks/capi/cluster.yml): nothing to make on the cloud:spec.controlPlaneEndpoint(the API's public name through the edges),status.initialization.provisioned, the zones asstatus.failureDomains, and the Secret<cluster>-kubeconfigCAPI reaches the cluster with (the cluster itself: thekubernetesService, a client certificate ofcapi_kubeconfig_hoursissued again with a third left).EthereConfig,EthereConfigTemplate(playbooks/capi/config.yml):spec.image; the bootstrap data is the image's user-data (provisioning/_userdata.yml, shared withprovisioning/servers.yml: nothing in it is specific to a server), in a Secret of its name it owns.EthereMachine,EthereMachineTemplate(playbooks/capi/machine.yml):spec.image,spec.flavor; the server has the resource's name, created with the Machine's bootstrap data (provisioning/_server.yml) in its failure domain or the least used zone, its cluster mode the annotationethe.re/cluster-mode(defaultjoin), enrolled through its console; a run interrupted resumes (the server found by its name).spec.providerIDis<provider>:///<UUID>, which the kubelet registers with (ethere-netconfig): CAPI finds the Node by it. The annotationethe.re/adopt: <server>reports an existing server instead. Its finalizer deletes the server and its ports (provisioning/delete.yml).-
EthereControlPlane(playbooks/capi/controlplane.yml,_hooks.yml,_machine_add.yml, its scale subresource):spec.replicas,spec.version,spec.machineTemplate.spec.infrastructureRef(anEthereMachineTemplate),spec.rollout.after. Stateless, one step per run, every minute, one at a time: an operator evicted in the middle (it runs on these servers: it was, by the drain, in the first rollout tested) goes on where the Machines are. Each run writes its status (the counters from the Machines' conditions,selector,versions,initialization.controlPlaneInitialized), then the first that applies:- a Machine without a controller owner claimed (the owner, the label
cluster.x-k8s.io/control-plane-name, the two hooks):capi/init.yml's; - a Machine being deleted: the hook CAPI waits for (its condition
Deleting's reason), one Machine at a time, nothing else meanwhile.provisioning/replace.yml's steps, around what CAPI does itself (the drain, the volumes, the Node's deletion).pre-drain.delete.hook.machine.cluster.x-k8s.io/ethere: the git writer moved if it's this one, to the oldest other replica of the current template (once in a rollout), else the oldest other one, the edges' name without it (k8s/gateway.yml) and the resolvers' caches; on qemu, the host's ports to another server.pre-terminate.delete.hook.machine.cluster.x-k8s.io/ethere: the hidden primary elsewhere serving the DNSEndpoints' records, then the server stopped and its etcd member removed, in the order the quorum allows (two members left: the member first, while it runs; a server already down: the member alone). The finalizer of itsEthereMachinethen deletes the server; - an unhealthy Machine (
OwnerRemediatedFalse: aMachineHealthCheckon the labelcluster.x-k8s.io/control-plane) deleted if the others hold the quorum: a dead member goes first (a fourth member couldn't join), then the count is short. A lost quorum isn't its: there is no API (etcd/recover.yml, an admin or a management cluster); - a Machine not Ready yet: wait;
- fewer than
spec.replicas: a Machine added (bootstrapif the cluster isn't initialized,joinotherwise; the least used failure domain); - the git names out of date (
git/writer.yml): a new replica's; one gone's with the next new one's, or at the rollout's end; - an outdated Machine (its
EthereMachine's image or flavor not the template's, its version notspec.version, older thanspec.rollout.afteror thanrotate_due_days: the servers' certificates): surge first, one more added, then, its etcd member answering, the oldest outdated deleted; more thanspec.replicaswithout one: the oldest deleted.
It rotates the control plane by age itself:
operator/reconcile.ymlleaves the images ofk8s_ctrlalone once anEthereControlPlaneexists. A server takes about 25 min on the test cloud, with 4 vCPU (about 8 to join, 5 for its git name and etcd member, 12 for its hooks and deletion). - a Machine without a controller owner claimed (the owner, the label
-
playbooks/capi/init.yml: over a cluster the playbooks made, in the operator's namespace: theCluster(itscontrolPlaneRef), itsEthereCluster, theEthereControlPlaneand its template, and each server of the etcd image adopted as a control planeMachine(labelcluster.x-k8s.io/control-plane, a Secret standing for its bootstrap data), which theEthereControlPlaneclaims. Workers can then beMachineDeployments.
Waits are the watches' periods (operator_watches): a playbook returns when something isn't there yet and runs again. Their objects are read through the admin's rights of the playbooks; status is written with kubectl patch --subresource=status (manageStatus: false), initialization and failureDomains never empty.
Waits are watch periods because the playbooks act with their own admin certificate: ansible-operator's dependent watches only see what goes through its proxy.
On the qemu provider (docs/admin/standalone.md, tests/standalone/standalone.test.js): the operator in the standalone's own VM, asking the host through the metadata service; a rollout with replicas: 1 takes about 20 min at 4 vCPU: a new VM joins (two etcd members), the host's ports move to it, the old one drains, its volumes follow their pods, its member goes while it runs, it stops.
The two modes¶
- Internal (a cluster managing itself):
capi/init.ymlas above, thenkubernetes.default.svcfor CAPI, the operator on a control plane server's host network, the edge on its host. - External (
operator_kubeconfig: a management cluster's kubeconfig;tests/cluster/external.test.js):capi/init.ymlfrom the workstation, over a cluster the playbooks made, installs the operator and declares the objects on the management cluster, in a namespace per cluster (ethere-<cluster>), and leaves nothing of the operator's in the managed cluster. AMachineHealthCheckcomes withcapi/init.ymlin both modes, for more than one server. There:operator_kubeconfigis where the CAPI objects and the operator are (in its pod, its ServiceAccount's), the admin kubeconfig the managed cluster (etcd, git, DNS);- CAPI's
<cluster>-kubeconfigand the operator reach the cluster throughapi.<zone>, the operator's git through the first of its addresses that answers (edge_by_name: a stopped edge stays in DNS until its replacement takes it out); - the operator is on the pods' network by default (
operator_host_network: false), its egress aCiliumNetworkPolicy(kube-apiserver,world;playbooks/operator/_network.yml), DNS64 and NAT64 for IPv4 edges; - the git bundle must be in S3 (
git_bundle_in_clusteris refused: the Secret would be in the cluster to recover).
Tested on the test cloud: a management cluster of one aio manages three, in two OpenStack projects with a member's rights; the servers adopted, the Secret issued again with CAPI still connected, the git writer's server stopped and replaced. Two things it does not test: the operator reaching OpenStack from the pods' network (the test cloud's catalog gives literal IPv4 addresses, unreachable from IPv6 pods: its operator is on the host's network, and a separate pod checks the policy for the API and the public name), and a real api.<zone> (a fake DNS server answers the name, tests/fixtures/cluster/outside_dns.py).
The git writer's server down, the PKI isn't read (operator/pki.yml goes on with pki_dir as it is, until the next run), so that the hook moving the writer gets its turn.
An annotation on an EthereCluster runs it at once (watchAnnotationsChanges), otherwise every operator_reconcile_period.
A lost quorum or a lost cluster, on request¶
From a management cluster, an admin's annotation on the EthereCluster (kubectl annotate etherecluster <cluster> ethe.re/recover=<anything>) runs playbooks/capi/recover.yml. It is never automatic: a network partition seen from the management cluster must not delete servers. provisioning/cluster.yml still asks Nova twice before it acts. The run:
- pauses Cluster API (the
Cluster'sspec.paused, theEthereControlPlane's annotation); - runs
provisioning/cluster.yml(a recovery with a member left, a restore without any). It refuses to make a new empty cluster (cluster_new, or no backup: an admin's decision) and leaves alone the credentials an operator has no right to make, which etcd keeps; - adopts the new servers (
capi/init.yml), deletes the Machines of servers gone without their hooks; - resumes Cluster API and removes the annotation last: a failed run goes on where it was, except for a restore interrupted before its server got the snapshot's URL (
TODO.md). The annotation gone means Cluster API resumed: the new Machines still have to becomeRunning.
DNS still gives a lost cluster's old edges until the rebuilt one publishes its own, so the playbooks use the first address of the name that answers, else Nova's addresses (edge_nat64_prefix on the pods' network; the same from a workstation). Only an operator on a management cluster reads the etcd backup: its S3 user's policy has one more statement (operator/credentials.yml). That backup holds every Secret of the cluster; the operator, which holds its CA keys, could already issue itself their access.
A restore goes back to the last backup: what changed since is lost.
The design¶
What CAPI brings¶
Its core controllers and ecosystem:
Machine,MachineSet,MachineDeployment: progressive replacement of machines when the image changes — the cattle rotation;MachineHealthCheck: automatic replacement of unhealthy nodes;- Cluster Autoscaler integration (
clusterapiprovider, scale from zero): a base for on-demand GPU nodes (gpu.md), the flavor choice and billing-aligned scale-down staying ours (driving replicas); clusterctl, conventions, tooling.
What ethere provides¶
One Ansible operator (Operator SDK), whose roles are shared with the bootstrap playbooks (ethere Ansible collection):
| CAPI contract | ethere resources | Responsibility |
|---|---|---|
| Infrastructure cluster | EthereCluster |
private networks, subnet pool, anycast ports, server groups, API endpoint (anycast / DNS), availability zones as failure domains, provider profile (providers.md) |
| Infrastructure machine | EthereMachine, EthereMachineTemplate |
server creation (ports per role network, volumes, image, flavor, AZ, server group), providerID, addresses; deletion with ports/volumes cleanup (playbooks/provisioning/delete.yml does it for the floating IPs and ports; no volumes today) |
| Bootstrap | EthereConfig |
the minimal user-data of ethere images (zero post-boot configuration): the same for every server of an image, the network read from the metadata service (sys-apps/ethere-netconfig), the cluster mode from the server's metadata, the kubelet's providerID openstack:///<UUID>, the sentinel /run/cluster-api/bootstrap-success.complete |
| Control plane | EthereControlPlane |
all-in-one or dedicated roles, etcd membership (join through anycast, removal of dead members), rotations |
- DNS choreography: rollouts don't wait for DNS TTLs by themselves; the DNS controller holds machine deletion through CAPI lifecycle hooks (
pre-drain/pre-terminateannotations on Machines, Runtime SDK) until the record change has propagated (dns.md). - Management cluster: CAPI needs one; the cluster manages itself (self-hosted). Two ways to get there, both driven by the ethere bootstrap (Ansible):
- Adopt, then rotate (preferred, to evaluate first in the spike): the bootstrap playbooks create the first cluster as today (possibly minimal), install CAPI and the ethere providers in it, then declare the existing state: an
EthereClustermarked as externally managed (cluster.x-k8s.io/managed-byannotation: CAPI reads it, creates nothing), andMachine/EthereMachineobjects bound to the existing servers byproviderID(the infrastructure provider only needs an "adopt" path: find the server, report it ready, create nothing). A rollout then replaces the adopted machines one by one with CAPI-native ones: adoption only needs to be good enough to be replaced, and after the first rotation the cluster is fully native. No temporary cluster, Ansible end to end, and the same path takes over an existing cluster or finishes a disaster recovery rebuilt by the playbooks. To validate first: how CAPI lets MachineSets/MachineDeployments take adopted (ownerless) Machines over through their labels; the control plane provider must accept existing etcd members and control plane nodes. - Temporary cluster (standard CAPI path): start a single-node k0s (
k0s controller --single: one static binary embedding containerd, no Docker/podman needed, unlike kind) in a throwaway instance on the target cloud or in a rootless local QEMU VM;clusterctl initwith CAPI and the ethere providers; apply the cluster manifests (the providers create the cluster);clusterctl moveto it; destroy the temporary cluster.
- Adopt, then rotate (preferred, to evaluate first in the spike): the bootstrap playbooks create the first cluster as today (possibly minimal), install CAPI and the ethere providers in it, then declare the existing state: an
- Provider images: the ethere providers (Ansible operator) are container images, to be published in a registry reachable by both the temporary and the target clusters.
- Why Ansible only: one implementation for bootstrap and steady state; idempotent
openstack.cloudmodules; the Ansible operator model (a role per reconciliation, finalizers,k8s_status) fits the provider contracts. OpenTofu is good for static declarative infrastructure but ill-suited to per-machine reconciliation in a controller (state and lock per machine, slow plan/apply on each event), and a second tool doubles maintenance. CAPO is not used because its model doesn't fit ethere's (anycast ports, role networks, Octavia by default, user-data).
Risks¶
- The infrastructure provider contract is demanding (status conditions, immutable templates, failure domains, deletion semantics); CAPO matured on many edge cases we will meet too.
- The contract evolves (v1beta1 → v1beta2): follow CAPI releases.
- CAPI's own end-to-end test framework is in Go; ethere tests through its Bun suite on the test cloud.