Skip to content

Standalone clusters: the qemu provider

A standalone cluster is an ethere cluster on one Linux host: its servers are QEMU VMs of the same images as on OpenStack (ethere-k8s-standalone, an ethere-k8s-aio with S3 inside, of count 1 usually), rootless with KVM. It serves as a management cluster, a development or CI cluster, a small production site. The playbooks are OpenStack's: the provider (server_provider) only changes who runs the servers.

How it works

  • The provider. server_provider: qemu (a cloud profile's, e.g. inventory/group_vars/ethere_standalone/, keyed by OS_CLOUD as OpenStack's). The playbooks reach the servers through the ethere_machine action (plugins/action/ethere_machine.py: list, ports, metadata, stop, start, delete, create, console, expose, its results shaped as Nova's and Neutron's) and playbooks/provisioning/machine/<provider>/ (_networks.yml: the networks' names for the user-data; _create.yml: the server). On qemu they run libexec/qemu-host.py on the host, locally or over SSH (qemu_host, qemu_host_command).
  • The host (libexec/qemu-host.py, qemu_host_dir, private): every VM is a systemd user unit (<qemu_host_units>-<server>.service), with its passt (…-passt.service) and the private networks' switch (…-switch.service); a VM's disk is an overlay of the image's qcow2, which is never replaced while a VM uses it (images/<image>-<sha>.qcow2, the current one named by <image>.current). Its NICs, in this order:
    1. the metadata service: QEMU's user network, restricted (169.254.169.254 alone, nothing else reachable), on which qemu-host.py serve-metadata answers each request with OpenStack's user_data, meta_data.json and network_data.json from the VM's machine.json: the metadata the playbooks set while it runs (enrollment, a restore's URL) is seen at once. It is eth0, which the image's base.network configures by DHCP before the user-data runs; ethere-netconfig then gives it a static address (waited for as degraded: link-local);
    2. the external network: passt (passt.env, qemu-host.py passt), the host's own addresses (IPv4 by DHCP, IPv6 by DHCPv6 and router advertisements), egress through the host's sockets, no gateway mapped to the host's loopback (--no-map-gw), the host's upstream resolvers (not a local stub). The exposed ports (qemu_exposed_ports) are forwarded by one VM's passt at a time (expose); a client's address is kept;
    3. the private networks (one NIC each, as on Neutron): QEMU's datagram sockets to the switch, a hub per network: a rootless L2 between the host's VMs, nothing leaving the host, anycast as on Neutron (a rotation runs two VMs side by side). A VM's sockets are under s/<id>/ (UNIX socket paths are short). The machine is a q35 with 8 PCIe root ports declared at start (a hotplugged disk needs a free one), the kernel's console on a serial port appended to serial.log (the enrollment reads its current boot), a QMP socket (an ACPI shutdown on stop).
  • Enrollment as on OpenStack (docs/admin/pki.md): libexec/enroll-console.py reads the serial logs through the host tool (ETHERE_QEMU_HOST_COMMAND), the certificates go back through the metadata.
  • DNS. The cluster's knot primary answers the public too: dns_primary_exposed_link (the profile's: the external link) makes it listen on that link's addresses (the user-data writes it, dns_primary_exposed_link_path), and the host forwards its port 53 there. No secondaries.
  • Rotation (provisioning/rotate.yml, replace.yml): a new VM joins (etcd with two members meanwhile), the host's exposed ports move to it (expose: the old passt releases them, the new one binds them: a short outage), then the old one drains (the DNS primary moves with its pod) and goes.
  • Volumes: the hostdisk CSI driver (sys-apps/ethere-csi-hostdisk, Go; inventory/group_vars/k8s/hostdisk-csi.yml), the default StorageClass hostdisk (WaitForFirstConsumer, one node at a time, ext4). A volume is a qcow2 file of the host (volumes/<id>.qcow2), attached to the VM running its pod: QMP's blockdev-add and device_add of a virtio disk on a free PCIe root port, its serial the volume's first 20 characters (/dev/disk/by-id/virtio-…, as Cinder's); detached with device_del once the guest released it. An attachment is kept in the VM's machine.json (given again at its next start). The driver's controller asks the host through the metadata service (POST /ethere/volumes/{create,delete,attach,detach,list}, answered by qemu-host.py serve-metadata for that VM: the cluster's volumes and VMs only, no credential; the pods have no way to the metadata service). Both services of the driver run on the host (systemd), in ethere-k8s-standalone alone: the controller (csi-provisioner and csi-attacher in ethere-hostdisk reach its socket) and the node service (registered by csi-node-driver-registrar); k8s/volumes.yml applies their objects. A volume outlives its VM: a rotation detaches it from the old one and attaches it to the new one with its pod.
  • S3 inside the cluster: SeaweedFS (inventory/group_vars/all/s3.yml, playbooks/k8s/s3.yml: its Helm chart 4.52.0 all-in-one in ethere-s3, its data on a hostdisk volume, its identities generated once into a Secret: the admin, and the registry limited to its bucket, which the chart's hook creates). k8s_s3_internal (the qemu provider) applies it, and k8s/registry.yml (which imports it) points Zot there: the Service's cluster IP (Zot is on the host's network), the registry's identity. Its image is embedded in ethere-k8s-standalone alone (277 MB; an aio's roles, host_vars): the registry needs this S3 to run, and containerd doesn't fall back to docker.io while the registry's address answers "no route to host". The recovery data (etcd's backups, the git bundles) still go to an external S3 when one is given (etcd_backup_s3_user); without one, losing the host loses the cluster, and the git bundle is a Secret of the cluster (git_bundle_in_cluster, kube-system/git-bundle): the operator's first clone, nothing to recover from.
  • The operator (docs/admin/operator.md, docs/dev/cluster-api.md) runs in the cluster it manages and asks the host through its VM's metadata service: qemu_host_command is libexec/qemu-host-metadata.py (set by operator/deploy.yml), which POSTs the tool's verbs to /ethere/host. A VM may ask list, console, stop, start, create, delete, metadata and expose, on this cluster's VMs: no SSH, no key, no route to the host. There is no credential either: whoever is root on a VM, or runs a pod on the host's network, can do the same (the pods have no way to it). It needs no OpenStack credentials, and no S3 user with the bundle in the cluster (below).
  • Not yet: the delegation of the public zones to the host (provisioning/dns.yml), a host over SSH tested.

The host

Any Linux host with:

  • KVM (/dev/kvm usable by the user: the kvm group), QEMU (qemu-system-x86_64 ≥ 7.2, qemu-img), OVMF (found in the usual paths, else qemu_ovmf_code/qemu_ovmf_vars), passt (the passt package), Python 3, systemd's user manager. standalone/host.yml checks them.
  • loginctl enable-linger <user>: the VMs keep running without a session, and start with the host.
  • The exposed ports below 1024 (443, 80, 53 by default): sysctl net.ipv4.ip_unprivileged_port_start=53 (persisted in /etc/sysctl.d/), or other ports in qemu_exposed_ports.
  • passt's AppArmor profile (Ubuntu, Debian) lets it write under the user's home and /tmp only: keep qemu_host_dir there (its default is ~/.local/share/ethere/qemu/).
  • A dedicated user when the cluster (its operator) manages its own host: never an admin's own account. Its VMs can create and delete each other through the metadata service (above), nothing else of the host.

Running one

# The cloud profile: server_provider: qemu, the external network, dns_primary_exposed_link (see ethere_standalone).
OS_CLOUD=<standalone cloud> OS_REGION_NAME=<region> \
  ansible-playbook $ETHERE_ROOT_DIR/playbooks/standalone/up.yml --limit localhost,ethere-k8s-standalone -e confirmation=y

standalone/up.yml runs standalone/host.yml (the prerequisites, the tool copied to qemu_host_dir, its settings, the topology's images copied in) then provisioning/cluster.yml (k8s/volumes.yml: the hostdisk driver's objects; k8s/s3.yml, then the registry on it). The workstation reaches the API through the host's exposed HTTPS port (edge_address, edge_address_port when it isn't the cluster's public name).

Test: tests/standalone/standalone.test.js (ETHERE_TEST_STANDALONE=1, on the workstation).