Operations¶
The procedures on a running cluster and its backups. Each is a playbook an admin runs from the workstation; the
operator runs some of them on its own (The operator), with the same playbooks: an admin can always
take over. Every command runs in the cluster's shell (etherectl shell), with OS_REGION_NAME the region of the
servers concerned.
Which procedure¶
| The situation | The procedure | By the operator |
|---|---|---|
| A server misbehaves, or must run a newer image or user-data | Replace a server | no |
| The servers age (their certificates last a week) | Rotate the servers | yes, every 30 minutes |
| Backups to set up, or their settings changed | Back up etcd | its CronJob, once set |
| Servers missing, stopped or in error; not sure what state the cluster is in | Reconcile the cluster | no |
| Two servers of three gone, the API down | Recover a lost quorum (through reconciling) | no |
| The cluster or its region lost | Restore the cluster (through reconciling) | no |
When in doubt, reconcile: provisioning/cluster.yml looks at what Nova shows and picks the procedure.
Replacing a server¶
ansible-playbook $ETHERE_ROOT_DIR/playbooks/provisioning/replace.yml --limit localhost,<its image> -e replace_server=<name> -e confirmation=y
A server is replaced by a new one, never repaired. One server at a time: a new server of the same image is added in
the same availability zone (servers_availability_zone: the zones stay spread), then the old one goes, each step
checked before the next; nothing is deleted until the new server does the old one's job. About 20 minutes for a
cluster server, 10 for a DNS secondary (dampetra on OVHcloud).
After a failure, run it again: it goes on with the server it added (the image one above its count) instead of
adding another. The playbooks reach the cluster through an edge that isn't the old server (edge_exclude).
What happens, for a cluster server (an ethere-k8s-aio: etcd, Kubernetes, git replica, edge):
- Its etcd member removed first if it is down: with one member of three dead, etcd refuses a fourth (the quorum would need three started).
- The new server created and enrolled (
provisioning/servers.yml, one more than the image's count,only_server_index: the others left alone), its nodeReady, its etcd member started and answering (etcd/members.yml,etcd_wait_member). - The git writer moved to another old replica if the old server is it (
git/writer.yml,git_writer_server): an old one is surely up to date, the new one may still be catching up. - The old server drained (only cordoned if its node is down: its pods never finish there), and the records the hidden primary served before served again (ExternalDNS publishes them again if the primary moved).
- It is stopped, its etcd member removed (
etcd/members.yml: refused while it runs,etcd-startwould add it back), it is deleted with its ports (provisioning/delete.yml,delete_down_ports: false: another stopped server's ports stay), its Node deleted, and the git names updated (git/writer.yml).
For a DNS secondary: the new one serves the zones on its public addresses, the delegation moves to it
(provisioning/dns.yml, the old one excluded: dns_secondaries_excluded), the parent zones' own servers give the
name servers' new addresses (not a resolver's cache), then the resolvers' caches expire (replace_dns_wait,
dns_provider_ttl + 30 s) before the old one is deleted. The other secondaries keep serving meanwhile.
By hand, the steps are the playbooks named above, in this order.
Rotating the servers¶
ansible-playbook $ETHERE_ROOT_DIR/playbooks/provisioning/rotate.yml --limit localhost,<image> -e confirmation=y [-e rotate_due_days=N]
A server's certificates last server_max_age_days (7) and nothing renews them in place (PKI): it is
replaced before then. The operator does it, every reconciliation, for every image of its region and of its
cloud's other regions. By hand (the operator down, or an image rolled out now):
- One run replaces the oldest server of an image created
rotate_due_daysago or more (defaultserver_max_age_days- 2: two days to spare), byprovisioning/replace.yml; none due, nothing is done. - Run it as many times as the image's count, in each region (dampetra: three times for
ethere-k8s-aioin EU-WEST-PAR, once forethere-dns-secondaryin GRA9 and in SBG5): the servers it added are the newest, never picked before the old ones. -e rotate_due_days=0takes the oldest whatever its age: an image rollout.- An image one server above its count (a replacement that failed) is refused: finish it with
replace.ymlfirst. A replacement interrupted (the operator's own server drained, an admin's run cut) is resumed first:replace.ymlmarks the server it replaces in Nova (The operator).
Backing up etcd¶
etcd holds the cluster's state: its objects, the DNSEndpoint records, the service account key. The rest (the PKI,
the tenants' applications through Flux) comes from git repositories, which have their own S3 bundles
(Git). k8s/configure.yml runs it too, from the workstation (it makes the backups' S3 user).
It sets a CronJob of the cluster (etcd-backup, namespace kube-system, every etcd_backup_schedule): its pod
runs on a control plane server (the host's network, its etcdctl and curl, the apiserver's etcd client
certificate) and uploads a snapshot to etcd_backup_key in the versioned bucket etcd_backup_bucket, the older
versions deleted after etcd_backup_retention_days (inventory/group_vars/all/etcd.yml).
- Its own S3 user, limited to that (
etcd_backup_s3_user: on OVHcloud an Object Storage user made by the playbook,ovh_s3_user, found again by its description; its keys in the Secretetcd-backup), never the admin's credentials. - Where: the cloud or region profile gives
etcd_backup_s3_endpointandetcd_backup_s3_regionas the cluster reaches them. Run it again after a change of these settings. - Now:
etcd_backup_nowmakes a backup at once and waits for it.
Reconciling a cluster¶
ansible-playbook $ETHERE_ROOT_DIR/playbooks/provisioning/cluster.yml --limit localhost,<etcd image> -e confirmation=y
One playbook for every case, the one that made the cluster. Nova is the only authority on which servers exist and run: it decides from what Nova shows for the cluster's region (its API failing stops everything) and whether a backup exists:
| Nova shows (the etcd image) | Done |
|---|---|
| no server, a backup | a restore (etcd/restore.yml, below) |
no server, no backup (or -e cluster_new=true) |
a new cluster |
| fewer running servers than a majority, some running | a recovery (etcd/recover.yml, below) |
| none running, some stopped or in error | those deleted, then a restore (or a new cluster) |
| otherwise | the missing servers added |
Then the git repositories (git/writer.yml; git/seed.yml after a restore or a new cluster), the Nodes of the
servers gone deleted (those not Ready only), and the cluster's configuration (k8s/configure.yml: the backups
among it).
- Destructive steps only on Nova's word, twice: before a restore or a recovery, Nova is asked again
cluster_observe_delaylater (120 s); anything different stops it. -e cluster_auto_recovery=falsestops before them, telling why.- A server stopped while the quorum holds is left alone: replace it (above).
Recovering a lost quorum¶
Two servers of three lost, one left: its member still holds the cluster's data, but refuses to serve without a majority (the API doesn't answer). Reconciling recovers it; by hand:
ansible-playbook $ETHERE_ROOT_DIR/playbooks/etcd/recover.yml --limit localhost,<etcd image> -e confirmation=y
Only when Nova shows fewer running servers than a majority (never because they don't answer: a network partition would make two clusters). Then:
- Those not running are deleted (several left in a larger cluster: all but the newest).
- The newest running one gets a request in its metadata (a nonce,
etcd_force_metadata_key) and a Nova reboot:ethere-etcd-restorereads it at boot (before the metadata block),etcd-startrestarts its member once with--force-new-cluster(its own data, the same cluster ID: at most its last entries lost), only if no cluster answers on the anycast address. - The missing members join it (
provisioning/servers.yml, orprovisioning/cluster.ymlfor all of it).
Nothing written before the loss is lost, unlike a restore. About 15 minutes (drills).
Restoring a cluster¶
A region down, or a cluster lost: everything is rebuilt, in another region if need be (one cluster, one region), from S3 and the admins' own credentials. Reconciling restores it when Nova shows no server; by hand, in this order:
- The PKI:
git/restore.yml(Git) into the cluster'spki_dir. - etcd:
provisioning/networks.yml, thenetcd/restore.yml --limit localhost,<etcd image> -e confirmation=y [-e etcd_backup_version=<id>](refused while the image has servers): the first server of the image is created in the cluster moderestore; once enrolled, it gets the snapshot's URL in its metadata (presigned, validetcd_restore_url_hours, never in the user-data), restores it (etcdutl snapshot restore) and starts a cluster of its own, the restored state with a new membership; the URL then leaves its metadata. - The other servers:
provisioning/servers.yml -e pki_cluster_sync=false(they join it; the PKI's repository has no replica yet:pki_diris the authority, as before a cluster's first seed), the cluster's other images. - The git repositories:
git/writer.yml(a new replica designated, theDNSEndpointobjects of replicas gone replaced), thengit/seed.yml(the new replicas getpki_dir, its S3 bundle uploaded again). Then the delegation if a DNS secondary was rebuilt too (provisioning/dns.yml). - The Nodes of the servers gone:
kubectl delete node(their pods go with them).
What was written after the backup is lost. What the restored state still holds of the old cluster: the Nodes of servers gone (step 5), the leases of apiservers and leaders gone (they expire), the pods scheduled on gone Nodes (removed with them), the backups' CronJob and Secret (they run again as they were). About 20 minutes of playbooks (drills).
Drills¶
Run on dampetra's production, three aio servers on OVHcloud.
A restore (2026-10-29: two servers stopped, the quorum lost, then all deleted; the 40 minutes of an S3 ownership
issue fixed since left out): the servers deleted 132 s, git/restore.yml 17 s, etcd/restore.yml 359 s (the API
answering again from its end), servers.yml for the two others 550 s, k8s/kubeconfig.yml 17 s, git/writer.yml
79 s, git/seed.yml 28 s: about 20 minutes, the API down from the quorum's loss until the restored member ran. A
ConfigMap written before the backup was there, one written after it not; the backups' CronJob ran again by itself.
A recovery (2026-10-30: the two oldest servers stopped, the API down; then provisioning/cluster.yml): Nova read,
asked again 120 s later, the request set, the two stopped servers deleted, the third rebooted: its member forced a
new cluster about 5 minutes after the start (the API back), then the git writer, the two missing members created
and joined, the backups: 910 s in all. A ConfigMap written after the last backup was there; the pods' NAT64 on the
new nodes about a minute after they were Ready. The replacements before it (replace.yml, batched enrollment): 815
to 856 s per aio server, against 873 to 965 s.