Skip to content

Running a cluster

An ethere cluster is Kubernetes on OpenStack, built from images of its own and run by its admins from a workstation, then by the operator it hosts. This tab is for them: what a cluster is made of, how to build one, how to keep it running. Whoever only deploys into a cluster reads the Users tab instead.

What a cluster is

  • Images, not configuration. Every server boots an Exherbo image built for its role, everything baked in: no Ansible against a server, no SSH, a read-only root. A server is never updated nor repaired: it is replaced by a new one, from a newer image (Operations).
  • Few OpenStack services. Nova, Neutron, Glance, Cinder and S3, nothing else: no managed load balancer, DNS or secret store. The cluster serves its own DNS zone (DNS) and balances its public traffic through it (short TTLs), its Gateway behind edges (Gateway).
  • A private network per service, reached through anycast. etcd's clients, the Kubernetes API, the git replicas…: each its network, each service one address shared by every server providing it. A new server finds its peers there; nobody configures them (Providers lists what an OpenStack provider must allow).
  • IPv6 only inside. The pods reach IPv4 through NAT64 (Egress), and start closed (Network policies).
  • Its own PKI. Short-lived certificates and monthly CAs, in a git repository the cluster replicates and backs up to S3 (PKI, Secrets, Git).
  • One cluster, one region. Another region is where it is rebuilt from its backups.
  • Two shapes. All-in-one servers (ethere-k8s-aio: etcd, control plane and node on each), with worker nodes (ethere-k8s-node) added on top; or dedicated roles (ethere-quorum, ethere-k8s-ctrl, ethere-k8s-node). On one host, the same images in QEMU VMs: Standalone clusters.

The workstation and the cluster's configuration

An admin works from a workstation (Host setup), in the cluster's configuration: a git repository holding the cluster's own settings (inventory/group_vars, openstack/clouds.yaml, secrets/ encrypted with sops and age, profile.d/) and the version of ethere it runs (ethere.yaml: ethere: {ref: <commit>}). Moving a cluster to a newer ethere is changing that ref, a commit of its configuration.

etherectl shell              # the cluster's shell: the pinned ethere, its tools, the cluster's credentials
etherectl ns <command>       # root inside a user namespace (the image builds): never sudo

Every playbook runs from that shell ($ETHERE_ROOT_DIR/playbooks/…). etherectl itself only manages the workstation (its admins, the test cloud, the namespace): what acts on a cluster is always a playbook, the same one the operator runs.

Building a cluster

In this order, each step a playbook:

  1. The images: downloads.yml, builder/setup.yml, then images.yml (built in the namespace, uploaded to Glance).
  2. The networks: provisioning/networks.yml.
  3. The servers and the cluster: provisioning/cluster.yml creates them and makes the cluster (etcd, the control plane, the git writer, the PKI's repository seeded), or tops up, recovers or restores an existing one (Operations); then applies its configuration, k8s/configure.yml: the Gateway and its certificates, volumes, the registry, etcd's backups, the tenants (Flux), the members' identities, this documentation. k8s/configure.yml alone applies it again.
  4. The DNS delegation of the cluster's zone to its name servers: provisioning/dns.yml (DNS).
  5. The operator: its image, operator/credentials.yml, operator/deploy.yml (Operator). From then on, it rotates the servers and applies the configuration by itself.

Keeping it running

  • Operations: replacing or rotating servers, etcd's backups, a lost quorum, a cluster restored. Each procedure is a playbook an admin runs by hand: the operator only automates them.
  • The operator: what it does and when, etherectl operator status and logs.
  • Tenants: the projects deploying into the cluster, their namespaces, names and members (a git repository Flux applies).
  • Identities: the members signing in (Dex, Pocket ID), their kubeconfig.
  • The registry: the cluster's images, the public registry the members push to.
  • Volumes: Cinder, the StorageClass.
  • The documentation: this site, served by the cluster.

What isn't done yet, and the designs ahead: the Developers tab (roadmap).