The operator¶
The cluster runs its own operations, so that it lasts months or years without anyone: the operator does on schedule what an admin does from the workstation (Operations), with the same playbooks and the same inputs. An admin can always do the same by hand, the operator down or not.
What it does¶
Every operator_reconcile_period (30 minutes), one reconciliation at a time (operator/reconcile.yml):
- The PKI's repository restored from its S3 bundle, the first time in a new pod (
git/restore.yml). - The servers' rotation of every image of
server_topologyin its region (OS_CLOUD,OS_REGION_NAME), then of its cloud's other regions, their region asOS_REGION_NAME(dampetra's DNS secondaries in GRA9 and SBG5, their delegation written withdns_provider_zones' credentials):provisioning/rotate.yml, the oldest server due replaced (Rotating the servers). The control plane's own servers are Cluster API's (Cluster API providers). - The cluster's configuration, last (a failure there never holds a rotation back):
k8s/configure.yml, but what needs an admin's credentials (cluster_credentials=false: the volumes', the registry's, the backups' credentials and the tenants' repository's Secret, which it can neither make nor decrypt) and the edges' name (k8s_gateway_edges=false: the replacements keep it).
A resource's annotation ansible.sdk.operatorframework.io/reconcile-period overrides the period.
What it doesn't do¶
- A lost quorum or a lost cluster: no API, no operator. The workstation runs
provisioning/cluster.yml(Reconciling a cluster), or an operator on a management cluster on an admin's request (Cluster API providers). - An admin's credentials: an admin runs
k8s/configure.ymlonce with them, and again to rotate them. - An admin's identity and recovery rights: no admin or recipient management, no provider root credentials, nothing deleting the backups or the bundles.
- Not yet:
provisioning/cluster.yml's top-up while the API answers (the missing servers added, the Nodes of servers gone deleted), the monthly CAs ahead of time, the rotation of its age key and of its derived credentials (roadmap).
Checking on it¶
etherectl operator status # the last reconciliation, step by step, and the operator's pod (--json: as it is)
etherectl operator logs --summaries # each run that changed something or failed, one line (--all: every run; -f)
etherectl operator logs -f # its whole log
The playbooks it runs are commands, whose output never reaches its log: operator/_runner.yml logs each run summed
up (What <playbook> did: the recap's counts, the changed and failed tasks by name; the filter playbook_summary,
plugins/filter/runs.py), and reconcile.yml writes its
account into its resource's status, failed or not (status.lastReconcile): started, finished, result
(Succeeded, Failed), failedTask, and a step per playbook it ran (git/restore.yml, k8s/kubeconfig.yml,
provisioning/rotate.yml <image>, k8s/configure.yml): its rc, changed, failed, changedTasks,
failedTasks. A reconciliation with nothing to do changes nothing (changed: 0 everywhere); the first one of a new
pod restores pki_dir and writes its kubeconfig.
Installing or updating it¶
Its image is one per ethere version: after moving the cluster to a newer ethere, the operator follows.
etherectl ns ansible-playbook $ETHERE_ROOT_DIR/playbooks/containers.yml --limit localhost,ethere-operator
ansible-playbook $ETHERE_ROOT_DIR/playbooks/registry/push.yml -e registry_push_source=ocidir://$ETHERE_BUILD_DIR/images/ethere-operator-oci:<tag> -e registry_push_repository=ethere-operator:<tag>
ansible-playbook $ETHERE_ROOT_DIR/playbooks/operator/credentials.yml -e operator_image=registry.ethere.internal/ethere-operator:<tag>
ansible-playbook $ETHERE_ROOT_DIR/playbooks/operator/deploy.yml -e operator_image=registry.ethere.internal/ethere-operator:<tag>
- Its image: built in the namespace (
containers.yml, its tag the build's time), pushed to the cluster's registry (Registry). - Its credentials (
operator/credentials.yml, from the workstation): an OpenStack application credentialethere:<cluster>:<region>:operator(aclouds.yamlunder the admin's cloud name: the cloud profile applies; every region of the cloud's topology), an S3 user reading and writing the git bundle and nothing else (ovh_s3_user: no deletion, the bucket not managed:git_bundle_bucket_manage: false), its age key (a Job of its own image runninglibexec/age-keygen.py: the Secretoperator-age, its recipient alone read back from a ConfigMap),dns_provider_zones' credentials sops-encrypted for it (the Secretoperator-dns, never in its image). Then the admin's side: a creation rule of the cluster's sops configuration for the CA keys (operator_sops_path_regex: the admins' recipients and the operator's), the CA keys re-encrypted (sops updatekeys),pki_dirpushed (git/push.yml). - Its Deployment (
operator/deploy.yml): the namespaceethere-operator, the CRDEthereOperations(ethe.re/v1alpha1), the framework's RBAC, the configuration, the cluster's, the Deployment and the cluster's resource (named after the cluster); nothing in the servers' images.
After any change of the cluster's inventory (the members, the registry's repositories…), operator/deploy.yml
again: the operator gets the cluster's configuration with its Deployment (the ConfigMap operator-cluster), and
would apply the configuration with the old one every reconciliation, undoing an admin's run (roadmap).
A change bringing new secrets the configuration generates in the cluster: run k8s/configure.yml from the
workstation before deploying the operator, so that one run makes them.
Its own server going away¶
A rotation replaces the oldest server, perhaps the one the operator runs on: the drain evicts it in the middle of
replace.yml, its Deployment starts it on another server. Nova keeps what was under way: replace.yml marks the
server being replaced in its metadata (replace_metadata_key) before adding the new one, and rotate.yml resumes a
marked server first, before or after its new server was added (an image one above its count without a mark is
refused; another replacement of the image is refused while one is marked). The same for an admin whose run was
interrupted.
How it works¶
Operator SDK's ansible-operator: a custom resource whose reconciliation runs the admin's playbooks. It runs a
playbook through ansible-runner, against an inventory it writes (localhost alone: none of ethere's nor the
cluster's variables), with the resource's name and namespace as extra variables (ansible_operator_meta;
watches.yaml: no inventory, no --limit). ethere's playbooks need theirs (--limit localhost,<image>, ethere's
inventory and the cluster's). So every kind's playbook is playbooks/operator/_runner.yml, which:
- reads the resource from the API (its ServiceAccount's token, straight to the API: the kubeconfig ansible-operator
gives names its proxy's default port whatever
--proxy-portsays); - runs the kind's playbook as an admin runs it:
ansible-playbook $ETHERE_ROOT_DIR/playbooks/<playbook>with ethere's and the cluster's inventory (ANSIBLE_INVENTORY, set for that run only: ansible-operator reads it as its runner's, a single path), the cluster's variables (ETHERE_OPERATOR_VARS:operator_spec, the ConfigMapoperator-config), the resource's spec, the whole resource asethere_resource,ethere_state=absentfrom a finalizer; without what ansible-runner sets for its own callbacks (ansible.cfgapplies, as on the workstation) nor ansible-operator's kubeconfig (its proxy and the ServiceAccount's identity: the playbooks make their own admin certificate).
reconcile.yml passes on the variables of reconcile_forwarded_vars it was given (the spec's: the edge, the
bundle's bucket, a test cloud's topology and buckets, rotate_due_days). The spec isn't converted to snake_case
(snakeCaseParameters: false: a region's name is a key of server_topology). One reconciliation at a time
(--max-concurrent-reconciles=1, a leader).
What it watches (operator_watches, all/operator.yml), one playbook each: its own resource EthereOperations
(operator/reconcile.yml) and Cluster API's providers (playbooks/capi/). Each kind's reconciliations at once are
MAX_CONCURRENT_RECONCILES_<KIND>_<GROUP> in its environment (ansible-operator takes them nowhere else:
concurrency). ansible-runner gives ansible-operator its events through ansible-runner-http: without it, every run
looks failed (did not receive playbook_on_stats event) and a finalizer never goes.
Two identities:
- Its ServiceAccount is the framework's only: the resource, its status, the leader's lease, events.
- The playbooks authenticate as an admin's do: a client certificate from the CA
k8s-rootmade byk8s/kubeconfig.yml(system:masters), the git signer's from the CApki, the registry pusher's from the CAregistry, etcd's client's; their CNsigner_name(ethere-admin-$USERon a workstation,ethere-operatorin its image:ETHERE_SIGNER_NAME). No RBAC list to keep in sync with the playbooks.
Its environment:
- On the control plane servers' host network (like Zot and the Cinder CSI controller; managing a cluster from a
management cluster, on the pods' network instead): the API's anycast address, the edge on its host (an aio
server's HAProxy:
edge_address127.0.0.1, the git writer and the API through it), the host's IPv4 egress (OpenStack's and S3's endpoints), the node's resolvers. Its framework's listeners moved off the host's ports (no metrics: 8443 is the Gateway's; the health probes and its proxy on localhost). - Its files:
pki_diranemptyDir, restored from the git bundle on its first run (git/restore.yml: no signer certificate before the CA keys are there), then following the writer as on a workstation (asafe.directoryof git: theemptyDir's root belongs to root);XDG_RUNTIME_DIR, its home andTMPDIRa memoryemptyDir(its temporary files, Ansible's and ansible-runner's, never on the node's disk), the Secrets mounted under it: itsclouds.yaml, its AWS credentials file, its age key; the cluster's sops configuration and the AWS configuration from a ConfigMap (operator-config). The age key generated by its Job in a memoryemptyDirtoo. - On the qemu provider (Standalone clusters): no OpenStack credentials (it asks its host through its VM's metadata service), no S3 user with the git bundle in the cluster.
Its image, one per ethere version (playbooks/containers.yml, the group containers): an Exherbo rootfs from the
image pipeline (images/_build.yml: Python 3.13 (the toolchain's), git, openssl, kubectl at the cluster's version,
sops: no age binary, sops reads age keys; helm and the charts k8s/configure.yml renders), ethere at
/opt/ethere-operator (ethere's files as git sees them (tracked, new and not ignored), the collections, ethere's
Python environment built in the chroot against its Python: uv sync --locked --group operator, ansible-runner
included, its watches.yaml), ansible-operator's binary (its release, pinned by sha256); assembled as an OCI image
layout of one layer (libexec/oci-layout.py) without a container engine. Nothing of a cluster: the same image for
every cluster and every cloud. The cluster's configuration comes with its Deployment: its tracked inventory/ and
ethere.yaml (operator_cluster_files) as the ConfigMap operator-cluster, mounted at /opt/cluster-config and
linked from /opt/cluster; its name, its own variables (operator_cluster_env: its profile.d's, from the shell
deploying it), the cloud, the region and the AWS profile as its environment.
Decisions¶
- Operator SDK's ansible-operator from the first increment (2026-11-01): a custom resource whose reconciliation runs the admin's playbooks; Cluster API's providers around the same playbooks.
- It signs with the monthly CAs' keys (2026-11-01), never an intermediate CA issued ahead by an admin (it will create the next months' CAs itself): a sops recipient of the CA keys, its own age key generated in the cluster (its private key never leaves it), its public key added by an admin.
- Its rights (2026-11-01): an admin's over the cluster's operation (the CA keys, the PKI's repository, the cluster's servers, the DNS provider's records, writing the backups and bundles, rotating its own derived credentials); not an admin's identity and recovery rights.
- The DNS provider's credential is a derived one, restricted to the zones' records (Secrets).
- Its image in the cluster's registry, pushed by an admin (
registry/push.yml). - The resource runs a wrapper (2026-11-02):
operator/reconcile.ymlruns the admin's commands as they are, rather than the playbooks adapted to ansible-runner's inventory. - Its image is an Exherbo rootfs (2026-11-02) built by ethere's image pipeline (its packages from paludis, like the servers'): no third-party base image.
Its history: the servers' rotation of its region first (2026-11-02, tests/cluster/operator.test.js: three aio
servers, rotate_due_days: 0: every one replaced by the operator, first scheduled on the oldest one: its own eviction, the replacement
resumed), the cluster's configuration (2026-11-16), the other regions' servers with their DNS delegation.