Running a cluster¶
An ethere cluster is Kubernetes on OpenStack, built from images of its own and run by its admins from a workstation, then by the operator it hosts. This tab is for them: what a cluster is made of, how to build one, how to keep it running. Whoever only deploys into a cluster reads the Users tab instead.
What a cluster is¶
- Images, not configuration. Every server boots an Exherbo image built for its role, everything baked in: no Ansible against a server, no SSH, a read-only root. A server is never updated nor repaired: it is replaced by a new one, from a newer image (Operations).
- Few OpenStack services. Nova, Neutron, Glance, Cinder and S3, nothing else: no managed load balancer, DNS or secret store. The cluster serves its own DNS zone (DNS) and balances its public traffic through it (short TTLs), its Gateway behind edges (Gateway).
- A private network per service, reached through anycast. etcd's clients, the Kubernetes API, the git replicas…: each its network, each service one address shared by every server providing it. A new server finds its peers there; nobody configures them (Providers lists what an OpenStack provider must allow).
- IPv6 only inside. The pods reach IPv4 through NAT64 (Egress), and start closed (Network policies).
- Its own PKI. Short-lived certificates and monthly CAs, in a git repository the cluster replicates and backs up to S3 (PKI, Secrets, Git).
- One cluster, one region. Another region is where it is rebuilt from its backups.
- Two shapes. All-in-one servers (
ethere-k8s-aio: etcd, control plane and node on each), with worker nodes (ethere-k8s-node) added on top; or dedicated roles (ethere-quorum,ethere-k8s-ctrl,ethere-k8s-node). On one host, the same images in QEMU VMs: Standalone clusters.
The workstation and the cluster's configuration¶
An admin works from a workstation (Host setup), in the cluster's configuration: a git repository
holding the cluster's own settings (inventory/group_vars, openstack/clouds.yaml, secrets/ encrypted with sops
and age, profile.d/) and the version of ethere it runs (ethere.yaml: ethere: {ref: <commit>}). Moving a cluster
to a newer ethere is changing that ref, a commit of its configuration.
etherectl shell # the cluster's shell: the pinned ethere, its tools, the cluster's credentials
etherectl ns <command> # root inside a user namespace (the image builds): never sudo
Every playbook runs from that shell ($ETHERE_ROOT_DIR/playbooks/…). etherectl itself only manages the
workstation (its admins, the test cloud, the namespace): what acts on a cluster is always a playbook, the same one
the operator runs.
Building a cluster¶
In this order, each step a playbook:
- The images:
downloads.yml,builder/setup.yml, thenimages.yml(built in the namespace, uploaded to Glance). - The networks:
provisioning/networks.yml. - The servers and the cluster:
provisioning/cluster.ymlcreates them and makes the cluster (etcd, the control plane, the git writer, the PKI's repository seeded), or tops up, recovers or restores an existing one (Operations); then applies its configuration,k8s/configure.yml: the Gateway and its certificates, volumes, the registry, etcd's backups, the tenants (Flux), the members' identities, this documentation.k8s/configure.ymlalone applies it again. - The DNS delegation of the cluster's zone to its name servers:
provisioning/dns.yml(DNS). - The operator: its image,
operator/credentials.yml,operator/deploy.yml(Operator). From then on, it rotates the servers and applies the configuration by itself.
Keeping it running¶
- Operations: replacing or rotating servers, etcd's backups, a lost quorum, a cluster restored. Each procedure is a playbook an admin runs by hand: the operator only automates them.
- The operator: what it does and when,
etherectl operator statusandlogs. - Tenants: the projects deploying into the cluster, their namespaces, names and members (a git repository Flux applies).
- Identities: the members signing in (Dex, Pocket ID), their kubeconfig.
- The registry: the cluster's images, the public registry the members push to.
- Volumes: Cinder, the StorageClass.
- The documentation: this site, served by the cluster.
What isn't done yet, and the designs ahead: the Developers tab (roadmap).