PKI¶
Where the PKI's state lives and how it is written, without post-boot configuration and without any service outside the cluster. Complements secrets.md. Node identity at boot (enrollment through the Nova console) is implemented. The PKI's repository is replicated in the cluster like the other git repositories: git.md.
What the repository holds¶
The PKI's state: the CAs (ca/<name>/<YYYY-MM>-cert.pem, the private keys sops-encrypted as -key.pem.sops, the bundles), and later the issued certificates worth keeping (revocation, audit). Today it is pki_dir ($ETHERE_CLUSTER_ROOT_DIR/secrets/pki) on the admin's workstation, created and used by provisioning/servers.yml.
Since node identity goes through the Nova console (below), nodes never write to the repository: no CSRs in git, no per-node paths. One kind of client writes, the signer (an admin's workstation today, the operator later), and the replicas copy. The CA private keys are only readable by the signer (sops, age recipients: the admins and the operator), so a replica, or a node compromised, never exposes them.
Bootstrap: the workstation first, then the cluster¶
The replicas run on servers whose certificates come from this PKI, so the first servers can't wait for it:
- Bootstrap: the workstation's
pki_diris authoritative, as today:provisioning/servers.ymlcreates the CAs (pkiincluded) and enrolls the first servers, the replicas among them. - Seeding: once the replicas serve, the signer designates the first writer (
playbooks/git/writer.yml,git.md), pushes the history ofpki_dirto it, waits for a majority of acknowledgements and uploads the repository's bundle to S3 (ansible-playbook $ETHERE_ROOT_DIR/playbooks/git/seed.yml; later writes:git/push.yml). - Steady state: the cluster's repository is authoritative.
pki_dirstays a clone, marked as such bygit/seed.ymlandgit/restore.yml(git config ethere.cluster true):provisioning/servers.ymlfast-forwards it to the writer'smainbefore signing, and pushes what it changed after (a majority of acknowledgements, then the S3 bundle), likegit/push.yml. - Disaster recovery: a cluster rebuilt from nothing starts again at 1, from the signer's clone or the newest S3 bundle (
git/restore.yml,git.md), whichever is newer.
The signer reaches the replicas from outside the cluster through the edge only (above).
Node identity at boot: enrollment through the Nova console¶
Decided on 2026-10-07 for every role (etcd, the control plane, the nodes), implemented by the enroll role (inventory/group_vars/enroll/, applied to every role in 00-hosts.ini) and provisioning/pki/_enroll.yml. Nothing secret goes through the user-data or Nova: private keys never leave the server. Keys and certificates live in /var/lib/ethere/enroll (<name>-{key,cert,ca}.pem, owned by the service using them), where etcd and the control plane read them.
Every server running a kubelet (nodes, control plane) enrolls two certificates: kubelet-client (O=system:nodes, CN=system:node:<server>, checked by the Node authorizer and NodeRestriction) and kubelet-serving (their K8s-Nodes-Net address in the SANs, checked by the apiserver through --kubelet-certificate-authority): no TLS bootstrap, no CSR approved inside the cluster, no CA key on the control plane.
The one secret shared by several servers, the Kubernetes service account signing key (a token signed by one apiserver must be accepted by the others), can't be enrolled: it is kept in etcd, like the cluster's Secrets. The first apiserver creates it in a transaction (only if still absent), the others read it (kube-apiserver-start, k8s_ctrl/services.yml); whoever holds an etcd client certificate owns the cluster anyway.
- At boot,
ethere-enroll.servicegenerates each key once in/var/lib/ethere/enroll(kept across reboots, gone with a rebuild), and prints one line per certificate on the console through the journal:ETHERE_ENROLL <name> <base64 DER CSR>. It prints them again every 30 s until it has its certificates: Nova only returns the end of the console log. - The signer (
provisioning/servers.ymlfor the servers it creates,provisioning/enroll.yml -e server=…for an existing one; later the operator) reads the console through the authenticated OpenStack API (openstack console log show). The console of an instance can only be written from inside it, and read by the project's API users: the request is bound to that instance. Only the requests printed since the last boot (the kernel's banner) count: after a rebuild, the log still holds the previous key's requests, which must never be signed again. - The signer checks the request's signature (possession of the key) and decides everything else itself: subject (
system:node:<server name>), extensions, IP SANs of the server's ports as Neutron knows them (pki_enroll_cert_dict__*); nothing the server put in its request is trusted. Lifetime:server_max_age_days(servers are replaced before their certificates expire: no renewal). The CA key only goes through openssl's standard input. - It returns the certificates, which are public, in the server's metadata: base64 DER in chunks of 255 characters (
ethere_enroll_<name>_<i>,ethere_enroll_<name>_n). The CA bundles, public too, come with the user-data. - The server reads
meta_data.jsonuntil each certificate matches its own key and its CA bundle, then installs it (a certificate of a previous key, e.g. before a rebuild, is ignored).
Trust boundary: a process able to print on a server's console is already root on it, hence already holds its keys: no escalation beyond that server. The metadata service (readable by the server's processes, pods included unless blocked) only holds public data. Requirement on the provider: the console log API (os-getConsoleOutput, see providers.md).
Lifetimes¶
- The CAs are monthly (
<name>/<YYYY-MM>), the servers trusting the previous, current and next ones (their bundles, from the user-data: fixed at the server's creation). The enrolled certificates lastserver_max_age_days(7) and are signed by the current month's CA, which lasts its month plusserver_max_age_days(pki_ca_not_after'soverlap_days,provisioning/pki/_ca.yml): a certificate signed on the month's last day doesn't outlive its CA.libexec/enroll-sign.pyrefuses to sign one that would. A chain fails once any of its certificates expired, its CA included: on 2026-11-01 the October CAs, which ended with their month, expired under certificates valid until the 7th, and the production cluster (etcd's peers, the apiservers, the kubelets) stopped accepting new connections. It was repaired in place: the servers rebooted (ethere-enrollfinds its certificates invalid and prints its requests again, with the same keys) and enrolled again (provisioning/enroll.yml), by the November CAs their bundles already trusted. - Nothing renews a running server's certificates in place, by design: a server lives less than
server_max_age_days, replaced before then (provisioning/rotate.yml,operations.md: by an admin today, on schedule by the operator later). Missed, it is rebooted and enrolled again (above). - Certificate lifetimes must leave enough time for a human to fix an outage before etcd's own certificates expire.