Skip to content

TODO

Build

  • Images per microarchitecture level: the images are built for x86-64-v2 (2026-10-24: the workstation's CPU, a Xeon E5-2430L, has no AVX2), which leaves the newer CPUs' instructions unused. Build variants for x86-64-v3 and x86-64-v4 (their own toolchain fingerprint and pbins, build_arch/build_goamd64 per variant) and pick the variant matching each OpenStack flavor's CPU; the QEMU boot tests then need a matching -cpu model, or a v2 workstation can only build them, not boot them.
  • Continuous delivery of the images (design in docs/dev/delivery.md): scheduled bump of the Exherbo submodules, build and tests (inspection, QEMU, test cloud), versioned and signed releases in every cloud's Glance, release channels (dev/beta/stable) in a releases git repository, clusters subscribed to a channel through Flux, nodes replaced through Cluster API with health checks and automatic halt. The pipeline runs on the cluster (Tekton or Argo Workflows triggered by Flux) on an on-demand bare metal builder node (KVM, local NVMe, pbins cache mirrored to S3), and every step stays runnable from a workstation. First steps: versioned images in Glance, a manual end-to-end "update" command, then the scheduler.
  • Immutable images, next steps: the erofs root and the systemd initrd creating /var are done. Remaining: a UKI (ukify, kernel + initrd + command line in one signed EFI binary) for Secure Boot, dm-verity on the read-only root (veritysetup generator in the initrd), finding the root through systemd-gpt-auto-generator instead of root=; containerd's btrfs snapshotter versus overlayfs on /var for the Kubernetes nodes.
  • Kernel built with Clang (hardening): the userland stays on GCC (glibc requires it, Clang is the less tested cc on Exherbo, no measurable gain), but the kernel can be built with LLVM=1 independently of the rest, which enables Clang-only hardening: kCFI (control-flow integrity of indirect calls) and kernel ThinLTO. LLVM is already in the builder (pulled by Rust): add clang and lld for builder/_kernel.yml. To evaluate: build time, compatibility with the kernel config fragments, boot (QEMU and Nova tests). Part of the toolchain: clang/lld versions go into toolchain_specs.
  • Report upstream (paludis/Exherbo): parts are not applied to binary packages. With paludis 3.0.2, */* parts: -documentation -development works for installs from source (checked with app-arch/lz4: no man page nor headers), but not with pbins: the pbin records the options (e.g. OPTIONS="parts:binaries parts:configuration parts:libraries" for dev-libs/openssl), yet its tarball holds every part and installing it (installbin) filters nothing. Reproduction: a pbins repository (binary_destination = true), the option in the environment, cave resolve -x --via-binary dev-libs/openssl into a chroot, then look for usr/share/man and include/ there. Once fixed, put the option back in images_base_paludis_options (the images' rsync filter does the job meanwhile).
  • arbor distfiles that can no longer be fetched (2026-10-25): app-misc/ca-certificates-20260816: Debian replaced it with 20260816+nmu1, and neither Exherbo's mirrors nor distfiles.exherbo.org keep the old .deb (Exherbo has no checksums to catch a different file). Worked around on the new workstation by fetching it from snapshot.debian.org by its SHA-1 (bb0fe1f8ed3f51e0ab6ec66bc4917af473f72299) into the builder's distfiles. Same day, dev-db/lmdb-0.9.36: git.openldap.org answers its archives with 429 Too Many Requests every time (an anti-bot wall), Exherbo's mirrors don't have it; taken from Gentoo's distfiles mirror (openldap-LMDB_0.9.36.tar.gz, checked against Gentoo's Manifest, same tree as github.com/LMDB/lmdb), recompressed to the .tar.bz2 the exheres expects. Fix: a bump to +nmu1 upstream (or on the ethere branch meanwhile); more generally, keep the distfiles (publish them to S3 with the pbins) so that a rebuild does not depend on upstream mirrors.
  • Merge request to arbor: providers_set for another root (commit on the ethere branch of gitlab.exherbo.org/voondo/arbor): it checked and selected the alternative under ${ROOT}, so a package calling it (e.g. sys-apps/systemd[efi]: ld bfd, compiler-tools binutils) died when built for an image chroot (--chroot-path): Alternative "bfd" for "ld" doesn't exist. Once merged, point the submodule back to exherbo/arbor.
  • Image build cache, next steps: the packages-level cache is done (images/_build.yml). Remaining: a fingerprint of the whole image (packages fingerprint, rendered configuration, addons, container image IDs, kernel, initrd, the playbooks' code) skipping an identical image entirely, including its erofs Minimize=best and qcow2 conversion; measure the remaining steps first (profile_tasks).
  • Cluster tests running concurrently: they share the private networks' names; a Keystone project per test (its own networks, quotas) would let two test files run at once on the test cloud.
  • QEMU boot tests failing fast: tests/images/lib/boot.js only detects a failed user-data; use images/lib/console.js (failed units, crash loops, kernel panic) like the cluster tests.
  • Catch more before the test cloud (most failures of the Kubernetes chantiers were findable without VM): static checks between rendered configuration and the extracted image (every hostPath of the rendered addons exists or lies under /var//run, Cilium's kernel requirements against the built .config, systemd-analyze verify of our units in the chroot, certificate SANs/subjects rendered and signed with _server/_ports fixtures, the rendering playbooks run on a fixture host); a control plane without VM (etcd, apiserver, controller-manager as processes in etherectl ns with a test PKI: flags, server-side dry-run of the addons, RBAC, the service account key in etcd); the embedded scripts (etcd-start, ethere-enroll, probes) run against simulated inputs. The test cloud would remain for the kubelet, Cilium/eBPF, Neutron anycast, the Nova console and memory.
  • Replacing and recovering, faster (docs/admin/operations.md: a replacement 815 to 856 s per aio server with the batched enrollment; a recovery 910 s, the API back after about 5): what is left is the server's creation and boot, waiting for the new node (~100 s), Nova stopping the old server (~100 s), the double observation of provisioning/cluster.yml (120 s, could be 60), the missing members created after the git writer (they could start while it runs: only the PKI's fetch needs it).
  • Test cloud settled after up: status() reports it healthy as soon as Nova's and Neutron's services are up, but for a few minutes after a boot creating a flavor, a subnet or a server fails at random (three cluster test runs failed differently right after an up). up should wait until a real operation succeeds (e.g. create and delete a flavor and a subnet).
  • pbins housekeeping: a command to delete the pbins repositories of old toolchain fingerprints ($BUILD/pbins/<fingerprint>/ are kept for rollbacks); one full world rebuild through the builder's pbins repository (pbins/builder/) so that rebuilding the builder from the stage compiles almost nothing (it only fills up with updates today); publishing pbins to S3 (binary_uri_prefix) for a CI or other machines.
  • Container image registry, next steps (docs/admin/registry.md, done 2026-11-02 on the test cloud: Zot behind a Service, its content in S3, mutual TLS for every client, the public registries cached; to deploy: dampetra's images rebuilt, the servers rotated, k8s/registry.yml): its garbage collection (several instances on one bucket: an upload in progress elsewhere must survive it), exposing it through the edge (done 2026-11-17: the public registry, registry.<zone>, a Zot of its own), deduplication (a shared cache: Redis or none), Spegel (in-cluster P2P mirror: nodes share layers, faster big pulls, e.g. CUDA images on GPU nodes). Hosts the ethere operator's images and the Cluster API providers'.
  • ethere's namespaces prefixed ethere- (decided 2026-11-01; ethere-registry the first; the Gateway done 2026-11-18: public in ethere-gateway, docs/admin/gateway.md): dns renamed on running clusters, never two DNS primaries (the anycast designates one): the old namespace's Deployment gone before the new one starts, the DNSEndpoint objects moved. Then k8s_gateway_legacy_namespaces removed, once every cluster has moved. cert-manager keeps its chart's name.
  • Volumes, next steps (docs/admin/volumes.md, done: the Cinder CSI driver, its default StorageClass, its application credential): CloudNativePG (PostgreSQL operator: streaming replication, Barman backups to S3) on Cinder volumes; volume snapshots (the snapshot CRDs and controller, the driver's snapshotter); local volumes (TopoLVM / local-path, much faster) only for what replicates itself and accepts losing a replica on every server replacement; the driver's credential derived and rotated by the operator.

Distribution

Discussed 2026-11-03, revised 2026-11-10. Done: ethere out of the clusters' configurations (one main clone ~/ethere, also the build workspace ETHERE_BUILD_DIR; worktrees per work stream; a cluster's pin ethere: {ref|path} in its ethere.yaml; etherectl shell; ethere's own cluster configuration project/; dampetra pinned, no submodule, 2026-11-11; the operator's image without any cluster's configuration). Order of the rest: tenants first (## Tenants), the workstation's standalone in ~/host, the laptop as an admin, then the public distribution below. Application projects (their own repositories, on this machine or another) never consume ethere: they deploy to a cluster as tenants.

  • dampetra's operator on the generic image (done in ethere: one operator image per ethere version, the cluster's configuration with its Deployment): rebuild the image, registry/push.yml, operator/deploy.yml with it (and operator/credentials.yml first if its credentials changed since).
  • ~/host pinned on path: ~/ethere: with phase 3 (the standalone), which makes it a cluster's configuration.
  • project/ethere.yaml names this workstation's admin and its age recipient (the cluster suites need an admin named $USER): generate it, or untrack it, before ethere goes public.
  • The laptop as an admin (and any machine): ~/ethere cloned, bin/etherectl linked, the cluster's configuration cloned; its own age recipient through etherectl admin (one key per machine, revoked alone), first tried on a Nitrokey 3 (age-plugin-fido2-hmac, FIDO2 hmac-secret; then age-plugin-yubikey, PIV: compatibility to check; sops 3.11 reads age plugins). Every decryption then asks for a touch: decrypt once per session into the runtime directory rather than ten times per playbook (the CA keys, kube-credential every 12 h). Images are never needed there (Glance by name), only to build or upload. key.enc is a plaintext age key today (docs/admin/secrets.md to say so).
  • Images from three sources, in order: local (the build workspace, while testing) → Glance (openstack image save, for a machine that didn't build, before any release) → public registries for each release. An etherectl images fetch (or a playbook) tries them in that order, verifies, fills images_local_dir; standalone/host.yml and the Glance upload stay as they are.
  • Public, versioned images per release (the public side of docs/dev/delivery.md's releases): signed OCI artifacts (the qcow2 as a blob, ORAS-style; cosign signature, a manifest image → sha256, size, ethere commit, toolchain fingerprint) pushed to several public registries (ghcr.io, quay.io…: no single provider), under a name prefix that is a variable: ethe.re's are the maintainer's releases, anyone building their own pushes under their own prefix. No hosting of ours. Before publishing: no image holds anything cluster-specific (secrets.test.js checks the key material), and GPL compliance: the corresponding sources published with each release (the distfiles of downloads/ and the builder's as OCI artifacts too, the exheres repositories' commits, public), the licenses of any firmware in the images.
  • ethere public: the repository (history audited first: no secret, nothing cluster-specific), the forks of the exheres repositories (already public).
  • Release archives: a signed archive per release (the code without paludis/repositories/), which a ref: that is a release version downloads instead of making a worktree; uv sync, bun install and the collections then run at its installation, read-only afterwards.
  • etherectl playbook <name> … for ansible-playbook $ETHERE_ROOT_DIR/playbooks/<name>.yml … (the docs spell the path out today).

Tenants

Discussed 2026-11-03 and 04: other projects (liveguard.io, aubonticket, medielife…) test on the local standalone cluster, then deploy to a production cluster, possibly with a staging namespace next to the production one. The cluster grants, the project deploys: a cluster knows what a tenant may use (namespaces, names, quotas, members), never what runs there (no chart, no version, no notion of environment). "Tenant" because "project" already meant the repository consuming ethere (now the cluster's configuration), and OpenStack's projects.

  • A member removed is deleted in Pocket ID (docs/admin/identity.md, found 2026-11-18): k8s/_pocket_id.yml creates and updates the users of k8s_identity_users, never deletes one; a member removed from it keeps signing in (with no rights left once their bindings are gone) until an admin deletes them in Pocket ID. Delete the users it doesn't declare (but its admins'), or disable them first. First increment done 2026-11-11 (docs/admin/tenants.md, tests/cluster/tenants.test.js): the ethere-tenants library chart and etherectl tenants init|update (the library copied as a directory in the declaration's charts/, not a .tgz), Flux as an addon reading the declaration from a git repository (k8s/flux.yml; decided 2026-11-11: Flux from the start, tenants/ in the cluster's configuration repository itself, e.g. dampetra's monorepo), the names policy (ethere-tenant-hostnames), the roles covering routes and Cilium policies, examples/tenant-app. Next, in this order: Dex (local accounts for the tests; dampetra-prod's connector to decide) and kubelogin, the apiserver trusting it through the user-data (--authentication-config); the registry per tenant (a prefix per tenant, exposed through the edge); then dampetra's own tenants/ and k8s/flux.yml on dampetra-prod. The items below are the design notes; what the increment settled differently is in docs/admin/tenants.md.

  • The declaration: one values.yaml per cluster in a git repository of its own, which is a Helm chart: Chart.yaml (pins the version of ethere's library chart ethere-tenants), values.yaml (the only file edited), a one-line template ({{ include "ethere.tenants" . }}), the library chart's .tgz committed in charts/ so that a rebuild depends on that repository alone. A library chart keeps the tenants at the top of the values, not under a subchart's key. Ansible doesn't read this file. No kustomize. The chart creates the namespaces, ResourceQuotas, roles, service accounts and the allowed names; validated by a schema (check that a library chart's values.schema.json applies, otherwise a check in the template). helm template . shows what will be applied, helm upgrade is the admin's way without Flux. helm.sh/resource-policy: keep on the namespaces: removing a tenant from the file must not delete its data.

    tenants:
      liveguard:
        namespaces:
          liveguard-staging: {hostnames: [staging.liveguard.io], quota: {cpu: "4", memory: 8Gi}}
          liveguard: {hostnames: [app.liveguard.io], quota: {cpu: "8", memory: 16Gi}}
        members:
          - {name: alice, role: admin, namespaces: [liveguard-staging, liveguard]}
          - {name: ci, role: deployer, namespaces: [liveguard-staging]}
    

    One release for every tenant at first (an error in one blocks the others); the chart can later generate one HelmRelease per tenant without changing the format.

  • Flux as an addon (already the choice of docs/dev/delivery.md), its images embedded like cert-manager's, its namespace in k8s_policies_exempt_namespaces: source-controller and helm-controller should be enough (to confirm: nothing else depending on kustomize-controller). A playbook (k8s/flux.yml) applies the entry point once: a GitRepository and a HelmRelease with chart: ./ and reconcileStrategy: Revision (the default, ChartVersion, shouldn't see a commit changing only values.yaml). Its multi-tenant lockdown patched in (--no-cross-namespace-refs, --default-service-account: not in Flux's default manifests). Measure its memory on a standalone cluster.

  • Where this git repository lives: anywhere, the cluster's own replicas included (decided 2026-11-04): the entry point takes any URL and its Secret, and the internal repository must work (dampetra will use it; docs/admin/operations.md expects it, the S3 bundles cover it). What the internal one needs: several repositories on the replicas (docs/admin/git.md, see "Several repositories on the git servers"), writing this repository as an admin does through the edge (git/push.yml only writes pki_dir today), its bundle to S3 and its restore, Flux on the control plane's host network to reach Git-Net, and a client certificate of the CA pki (to confirm: GitRepository with tls.crt/tls.key). An external forge with a deploy key can validate the rest of the chain before that work is done. In any case not ~/host as it is (Flux clones the whole repository, its journal included): a repository of its own or a git subtree.
  • Renewing Flux's client certificate: the CAs rotate monthly and nothing renews a certificate held in a Secret yet. A job or the operator; required by the internal repository (the CA pki), and needed as well if Flux reads Zot with mutual TLS.
  • Names: close the Gateway's hole, independently of the rest. The HTTPS listeners accept routes from every namespace (allowedRoutes: from: All, playbooks/k8s/gateway.yml) and ExternalDNS publishes each accepted route's names: with namespace-scoped deployers, a tenant could claim another one's name. One ValidatingAdmissionPolicy for the cluster (no webhook, stable since Kubernetes 1.30): the chart writes the allowed names in an annotation of the namespace (ethe.re/hostnames), the policy compares each HTTPRoute to it through namespaceObject. The roles must be admin, not cluster-admin (the default of flux create tenant): bound in a namespace, the latter should let a member edit that namespace, hence the annotation (to test).
  • Namespace-scoped kubeconfigs: k8s/kubeconfig.yml only makes admins' today, its credential plugin needs the cluster's PKI. A project writes into its namespaces (helm upgrade, or its own Flux objects), so a kubeconfig per member and cluster, with no cluster-wide right. To decide: how a member's identity is issued and renewed (a ServiceAccount's short-lived tokens, short-lived certificates from an admin, OIDC later).
  • A tenant's rights in the registry: done 2026-11-17 by the identity provider's groups (docs/admin/registry.md, the public registry: k8s_registry_public_repositories, a pattern per group). Left: deriving them from the tenants' declaration (a namespace's members, its prefix) rather than a list of its own; a Valkey failover test (its primary's pod deleted during a push); Valkey as a data service for the tenants (after CloudNativePG).
  • Names outside the cluster's zones (app.liveguard.io): their listener and their provider's records stay in the inventory (k8s_gateway_hostnames, dns_provider_zones, k8s/gateway.yml). To evaluate: a ListenerSet made by the chart (Gateway API; Cilium's support to check) for the listener; the provider's records stay a playbook's.
  • A project's side (its documentation now served by the cluster: docs/, docs/admin/documentation.md; what is left): its own targets (local, staging, prod) and values per target in its repository, its way of deploying in its namespaces (a script with helm upgrade to start with; its own OCIRepository and HelmRelease later, with the service account the chart gives), a promotion copying the image by digest from one cluster's registry to the other's (registry.ethere.internal/<name>@sha256:… is the same reference on both), so no rebuild between targets. Who may reach production stays the cluster's: the members' namespaces. What differs from a k3s: HTTPRoute instead of Ingress, listening on IPv6 (::), the default-deny policies (CiliumNetworkPolicy with fromEntities: [ingress], toFQDNs), images pushed to the registry, the default StorageClass left empty (hostdisk or cinder).
  • CloudNativePG on the clusters (see "Volumes, next steps"; liveguard needs it): cluster-wide, so installed by the cluster rather than by a tenant, and available on the standalone shape too (on hostdisk).
  • A standalone cluster's certificates: its names come from its zone (dns_zones, the cluster's), as on any cluster. Let's Encrypt's DNS-01 needs that zone delegated to the host, which docs/admin/standalone.md lists as not done (and, on the host's side, port 53 reachable from the Internet, a stable address). Without a delegation: a private ACME server or certificates made by hand (k8s_acme_server, k8s_acme_ca_bundle), trusted by each client device.
  • Same images on the local cluster and in production (or the next ones): otherwise a test passing locally says little about production. The channels of "Distribution" and docs/dev/delivery.md answer it: the local cluster on dev or beta, production on stable.
  • Considered and set aside:
    • a CRD applications.ethe.re with a reconciler: the values and Flux need no controller nor image rebuild. To reconsider if the registry's rights and the external names call for a reconciliation, or for a status; its spec would take the same values.
    • Kyverno: four Deployments and a webhook for what a native policy and a chart do (docs/admin/network-policies.md already avoids it).
    • Capsule (read at its main, 2026-11-03): it covers the model (Tenant, owners, HTTPRoute names per namespace, a quota across a tenant's namespaces), but its syntax was found heavier and its default webhooks are failurePolicy: Fail, among them one for every write in a tenant's namespaces and one for namespaces cluster-wide, with one replica: an outage risk during rotations, especially on a one-VM cluster. To reconsider if a quota per tenant becomes necessary (ResourceQuota is per namespace) or members should create namespaces themselves.

Provisioning and operations

  • Provider credentials from one root secret per admin (design in docs/admin/secrets.md; first adapter done: ovh_s3_user, the etcd backups' S3 user, its keys kept in the cluster's Secret only): Ansible adapters, one module per capability and provider (no OpenTofu, like the DNS delegation), derive scoped credentials from each admin's root credentials — OpenStack application credentials per component (openstack.cloud.application_credential), per-bucket S3 users (etcd backups, CNPG/Barman, Zot; ovh_s3_user on the OVHcloud API) — found by name (no state), sops-encrypted in the same run they are created, for the admins and the operator, which rotates them with its own scoped credential.
  • DNS delegation, next steps (docs/admin/dns.md, "Who manages what"; provisioning/dns.yml, done for dampetra: cloud.dampetra.com, two secondaries in GRA9 and SBG5, IPv4 only): adapters for other DNS providers as needed; DS records through CDS/CDNSKEY where the registry supports it (check for the cluster's TLDs), otherwise through the operator; the operator taking over the NS names' A/AAAA records (the TTL choreography of a replacement).
  • Reading the git repositories from outside: the edge routes the writer's and each replica's name only (docs/admin/git.md); when an outside reader needs the replicas' anycast, route a public name to it and add that name to the replicas' serving certificates.
  • More internal names in the internal zone ethere.internal (the git names are there, docs/admin/dns.md): the servers' names, today in /etc/hosts from the user-data.
  • The bundles' bucket written with a derived S3 user (docs/admin/git.md): git/seed.yml/git/push.yml upload with the admin's AWS profile today; an S3 user limited to the bucket (the derived credentials of docs/admin/secrets.md), and recovery (git/restore.yml) keeping the admin's own root credentials.
  • Applications' DNS records through ExternalDNS (docs/admin/dns.md; the Gateway's routes done, docs/admin/gateway.md): an ACL of the primary scoped to the applications' names (ExternalDNS's ACL allows any A, AAAA, CNAME and TXT of the public zones today).
  • Several repositories on the git servers (renamed ethere-git-server/git_replica/Git-Net on 2026-10-10, docs/admin/git.md): the secrets kept outside Kubernetes first (the cluster's secrets repository of docs/admin/secrets.md, cluster-secrets, sops-encrypted, the derived provider credentials among them, e.g. the S3 users, the bundles' bucket's included; the operator writes it like the PKI). To decide: paths and access per repository (which identities read and write which one), one writer for all or one per repository, and which CA the git servers' and their clients' certificates come from (today the CA pki, which stays the PKI's with pki_dir). The recovery of the repositories (the S3 bundles) must not depend on them: a rebuild reads the bucket with the admin's own root credentials, never with credentials stored in the repositories it is recovering.
  • DNS, next steps (docs/admin/dns.md): public IPv6 for the secondaries (OVHcloud's GRA9 and SBG5 have a dual-stack Ext-Net without DHCPv6: its IPv6 configured statically, then published); NOTIFY (from the NS records or the secondaries' requests) if the SOA refresh latency matters; a scoped TSIG key for cert-manager (knot ACLs, validated in the POC); DS records; the update latency to the secondaries measured on OVH.
  • Pod egress, next steps (NAT64/DNS64 and native IPv6 done, docs/admin/egress.md): kubernetes-sigs/nat64 has no release since v0.4.1 (July 2025): follow its main, or package a newer tag; filtering the egress (Cilium toFQDNs policies) once workloads need it; 464XLAT if a workload needs IPv4 literals.
  • External access to the API, next steps (done 2026-11-03: the public name api.<zone>, the credential plugin): the addons applied from outside (provisioning/cluster.yml, the operator) instead of baked into the control plane's image.
  • The Gateway, next steps (docs/admin/gateway.md, done: Cilium's Gateway on the aio servers behind their edge, cert-manager, the names): dedicated ethere-edge servers forwarding to the edge nodes' Gateway (they have no Envoy: only aio servers serve it today); the edges' name kept up to date by the operator (an edge failing stays in it until k8s/gateway.yml runs again); the records of names removed from k8s_gateway_hostnames deleted from their provider zone; a scoped TSIG key for cert-manager; the HTTP-01 path if ever needed (DNS-01 only today).
  • Port security on every network: today only the external ports get a security group (one per image, servers_security_group_rules__*); the private networks have port security disabled, which anycast and Cilium's native routing rely on, and which many providers forbid (docs/admin/providers.md). Instead: allowed address pairs for the anycast address on every carrier port (extend anycast.test.js to this mode, failover included) and for the pods' prefix on the K8s-Nodes-Net ports; per-network rules (etcd 2379/2380, K8s-Nodes-Net 6443/10250 and pod traffic, DNS-Net 53 and 853/udp), the same mechanism extended to one security group per image and network. Check first that security groups and address pairs apply to OVH's vRack ports (by hand, like the anycast check of 2026-10-26: carriers declaring the anycast address as an address pair, failover included), and add Git-Net (the git port) and the neighbor discovery's ICMPv6 to the rules. Then the production servers' private ports move to port security (a migration: the existing ports have it disabled). Note: an existing server's security groups are reconciled by openstack.cloud.server with the list it is given (_server.yml), which then covers every port of the server.
  • Pods' first connections while a cluster bootstraps: the check pods of the cluster tests fail to reach the API Service (fd00:baba:dada:ffff::1,443) for their first ~20 s on the worker nodes, never on the control plane, then succeed. Not a fresh node's matter (checked 2026-10-28: a worker replaced in a running cluster by provisioning/replace.yml, the other nodes' routes to its podCIDR there the second it was Ready, a pod on it reaching the API Service from its first attempt; the link-local addresses changed nothing either), but of the first minutes of a cluster (the tests' pods start then). Low priority: look at the apiservers' endpoints and the agents' state then, if it ever matters outside the tests.
  • Time synchronization on the test cloud: on OVHcloud it works (2026-10-27, dampetra: systemd-timesyncd synchronizes at boot with 0.exherbo.pool.ntp.org over IPv4, the first synchronization on the console, a step of ~0.3 s; the aio servers within a second of the workstation, the secondaries' events consistent with the primary's), but on the test cloud no synchronization shows (no reachable NTP server; the clock from the hypervisor, kvm-clock): give it an NTP source. Certificates issued on the fly stay backdated (by 10 minutes: utils/_signer_certificate.yml); on the test cloud (2026-10-25) the servers were about 4 s behind the workstation, enough to refuse a certificate issued at once. Later: NTP servers in the cloud profile (NTP=) rather than the public pool.
  • The servers' rotation by the operator (provisioning/rotate.yml, docs/admin/operations.md; decided 2026-11-01: no renewal in place, a server lives less than server_max_age_days): the operator runs rotate.yml on schedule for every image and region; until then an admin, at least weekly. A warning a few days before a certificate expires, from outside the cluster (it must work when the operator doesn't: the console, the probes). Also check the CAs' rotation lookup: pki_ca_rotation(offset=-1), pki_ca_rotation_previous, is the next month (and +1 the previous one): harmless while the three are used together.
  • The operator signs with the monthly CAs' keys (decided 2026-11-01; required: the cluster runs on its own for months, years, no admin needed, which rules out an intermediate CA issued ahead by an admin): the operator is a sops recipient of the CA keys (its own age key, docs/admin/pki.md), creates the next months' CAs itself and signs the enrollments and its git signer certificate with the same playbooks as an admin (provisioning/pki/_ca.yml, _enroll_sign.yml, utils/_signer_certificate.yml), the keys decrypted in its pod's memory only. Its age key lives in the cluster (a Secret, i.e. etcd: whoever holds etcd owns the cluster anyway, as for the service account key); the admins keep the same power (break-glass). Its age key is generated in the cluster (decided 2026-11-01: the private key never leaves it), its public key added as a recipient by an admin (etherectl admin, ethere.yaml); to decide: its rotation.
  • The operator's rights (discussed 2026-11-01): an admin's over the cluster's operation, since it must run alone for years: the CA keys and the monthly CAs, the PKI's repository, the cluster's servers (create, delete), the DNS provider's records (glue, NS), writing etcd's backups and the git bundles to S3, rotating its own derived credentials (docs/admin/secrets.md). No more power than today: whoever holds etcd owns the cluster already. Not an admin's identity and recovery rights: no admin or sops recipient management (ethere.yaml, etherectl admin: a compromised cluster would take the future secrets or lock the admins out), no provider root credentials (billing, projects, IAM: derived credentials scoped to the project and the buckets only), and nothing that destroys the recovery path: its S3 credentials write the backups and bundles but can't delete them nor overwrite their versions (versioning, object lock), so a compromised or broken cluster can't erase what rebuilds it; the recovery reads the bucket with an admin's own credentials (already the rule for the git bundles, above).
  • The cluster's documentation, next (first increment done: docs/admin/documentation.md): a personalised view over HTTP (the caller's token, SubjectAccessReview, the same ConfigMaps) with the members' identities (Dex); a page per admission refusal; kubectl explain on the operator's CRDs; the onboarding through it end to end (an application handed only the URL).
  • The documentation's content by audience (the site by audience done 2026-11-18: docs/ as its tree, Zensical, docs/admin/documentation.md; rewritten as guides: the Admins and Developers tabs' entries, operations.md, operator.md, tenants.md, identity.md). Next: the Admins tab's other pages as guides (Components: registry, gateway, dns, egress, network policies, volumes, pki, secrets, git; Setting up: host-setup, providers; standalone), their design notes and dated decisions in a section of their own or the Developers tab; the code paths they name linked to the repository at the version served; Zensical's admonitions (the warnings buried in the text: a namespace kept with its data, a member removed kept by Pocket ID) and content tabs, the CommonMark comparison of etherectl docs check taught to skip them; Zensical followed out of its 0.x (pinned: pyproject.toml).
  • An MCP server in the cluster (idea, 2026-11-01; with the documentation above): agents deploying into the cluster connect to it (Model Context Protocol, over HTTP behind the Gateway). Resources: the documentation of the version running. Tools reading the cluster live: what a namespace may reach (its network policies), the Gateway's names and how to get one, the StorageClasses and zones, the quotas; a manifest checked before it is applied (a server-side dry run, the refusals explained with the page concerned). Authenticated as the Kubernetes identity calling it (its token passed through, the API's own authorization: no rights of its own), read-only by default. To decide: an existing Kubernetes MCP server extended or our own, and whether it may apply anything (then only what the caller's identity may).
  • Kubernetes control plane, next steps: withdraw the API anycast address when the apiserver becomes unhealthy, not only when it stops (a health check timer); rotation of the service account key kept in etcd (publish the next key, accepted by every apiserver through several --service-account-key-file, then switch the signing key); --cluster-signing-* stays off until node identity is settled (docs/admin/pki.md).
  • etcd 3.7 with Kubernetes 1.37 (1.36 is validated with etcd 3.6; 3.7's RangeStream needs 1.37's EtcdRangeStream feature gate): a rolling upgrade needs every member on ≥ 3.6.11 (done: 3.6.15), and 3.7 refuses to start with any --experimental-* flag (none used today). Bump etcd_version and the exheres in paludis/repositories/ethere/.
  • The operator follows the cluster: the ethere it pins and its inventory (found 2026-11-18, to discuss: a structural issue). The operator applies k8s/configure.yml every reconciliation with its own image's ethere (its version names the documentation server's image, its playbooks make the objects) and the inventory it was deployed with (operator/deploy.yml: the ConfigMap operator-cluster). So an admin's run from the workstation with a newer ref or a changed inventory is undone within 30 minutes: on 2026-11-18 it put dampetra's documentation server back on its older image three times, after rollouts made from the workstation only; a user's groups would be put back the same way. Until fixed, a cluster moving its ref, or changing its inventory, rebuilds the operator's image at that ref and runs operator/deploy.yml (docs/admin/operator.md). Directions: the operator reads the cluster's ethere.yaml and inventory from its configuration repository (the tenants' GitRepository already brings it: a commit pushed is applied, documentation included), and either fetches that ethere (a worktree at the ref, as etherectl makes them) or refuses and reports the gap (its status) while its image isn't at the same ref; an admin's k8s/configure.yml refuses, or warns, when the operator's version differs from the workstation's.
  • Ansible-based Kubernetes operator (docs/admin/operator.md; first increment done 2026-11-02: the servers' rotation of a region, operator/reconcile.yml, then the cluster's configuration, k8s/configure.yml, 2026-11-16; next: provisioning/cluster.yml in its loop, the cluster's configuration read from its git repository rather than the Deployment's ConfigMap (a change of its inventory without a redeployment), the monthly CAs ahead, the rotation of its age key and derived credentials, the image's own rollout (built and pushed by hand today); decided 2026-11-01: Operator SDK's ansible-operator from the first increment, a CRD whose reconciliation runs the admin's playbooks, cluster.yml and rotate.yml first; the DNS provider's credential a derived one, restricted to the zones' records, docs/admin/secrets.md; the container registry first, below): instance rotation, DNS records on the hidden primary (knot), CA/certificate/secret rotation, removal of dead etcd members (by hand today: etcd/members.yml, docs/admin/operations.md). Move the logic shared with the bootstrap into roles of an ethere collection. Direction: implement Cluster API providers (infrastructure, bootstrap, control plane) with Ansible only, on top of CAPI's core (docs/dev/cluster-api.md).
    • One reconciliation playbook, the loop's body (provisioning/cluster.yml, a first version 2026-10-29: one etcd image, a stopped server while the quorum holds left to replace.yml; to do: several images (quorum + ctrl + node), the replacements, the DNS secondaries, the delegation): it observes (Nova, authoritative for which servers exist and run; the consoles' ETHERE_ETCD lines and the API when it answers; the backups in S3), compares with server_topology, and converges by calling the playbooks a human runs (servers.yml, replace.yml, etcd/members.yml, etcd/restore.yml, the next one): top up missing servers, replace failed ones, recover a lost quorum, restore from a backup, reseed the git repositories, delete the Nodes of gone servers, set the backups. A human runs the same playbook; the specialized ones stay its building blocks.
    • Destructive transitions only on Nova's word, then automatic (discussed 2026-10-29): forcing a new etcd cluster or restoring from a backup only once Nova (checked every time, for the cluster's project and region) shows the members' servers gone, those stopped or in error deleted by the playbook first: never because they don't answer (a network partition would make a second cluster: split brain). An API error or an incomplete answer means "unknown": nothing done, reported; the observation confirmed twice, minutes apart. Then by default: a member left, --force-new-cluster; none, a backup, the restore. Flags to prevent or choose rather than to allow: cluster_auto_recovery: false (the operator waits for an admin), cluster_new=true (a new empty cluster despite a backup).
    • A lost quorum with a member left (etcd/recover.yml, done 2026-10-29, called by provisioning/cluster.yml): the surviving member restarted once with --force-new-cluster (its own data: at most its last entries lost, against up to etcd_backup_schedule from S3), triggered through the OpenStack API like the restore (a flag in its metadata, then a Nova reboot: read before the metadata block, etcd-start honours it once), then the missing members replaced. About two minutes instead of a rebuild's twenty.
    • The operator without the API: when the quorum is lost the API doesn't answer, and an Operator SDK operator loses its leader lease and stops; its recovery path (Nova, the consoles, the metadata) must not depend on the API nor on a lease (or another leader election, e.g. through Nova).
    • Total loss: nothing in the cluster survives to act; the reconciler then runs outside it (the workstation, a CI, or a management cluster in another region: Cluster API's model, docs/dev/cluster-api.md) with the same playbook.
    • A restore interrupted before its server got the snapshot's URL: that server waits for the URL with no data, and the next run, seeing servers, doesn't give it (the cases after it resume: tests/cluster/external.test.js). Read from the code, not run. An admin deletes the servers and runs it again.
    • A restore on request from a management cluster, on OVHcloud: etcd/restore.yml presigns the snapshot's URL for a version with the operator's keys, whose policy can't hold s3:GetObjectVersion there. Tested on moto only, which checks no policy: try it on OVHcloud's S3, and if refused, presign without a version.
    • Two first clones of pki_dir at an operator's start: operator/reconcile.yml's git/restore.yml runs outside capi/_prepare.yml's lock; on OVHcloud at the first start, once, git clone found the directory just made by the other run. The next reconciliation succeeded.
    • The primary's pod moving under git/writer.yml: its waits kubectl exec the pod found at their start; when that pod goes (a rollout right after a recovery, once in tests/cluster/external.test.js), they run to git_writer_timeout on NotFound, about 5 minutes, and the next reconciliation succeeds. Each attempt should find the pod again; exec deploy/dns-primary isn't enough: it can pick a pod being deleted on a stopped server.
  • Cluster API spike (docs/dev/cluster-api.md): on the test cloud, minimal EthereMachine/EthereMachineTemplate + EthereConfig in an Ansible operator, a MachineDeployment making an ethere-k8s-node join, scale down and rollout; then adoption of existing servers followed by a rotation (preferred bootstrap path, no temporary cluster); assess contract coverage and reconciliation latency.
  • Two failures once, passing again (2026-11-10, a full tier run at three jobs, the images just rebuilt): registry.test.js's Zot DaemonSet scheduled on 1 aio server of 2, the cleanup's memory report listing one aio server only (a server missing, not checked: servers.yml doesn't verify the count once its creations are asked together); k8s-aio.test.js's third aio server without an ETHERE_ETCD joining line on its whole console though its node was Ready. Both passed when run again at once.
  • A standalone cluster's primary restarted empty: it answers the public itself (no secondaries), so a restart still serves the zones' base alone until ExternalDNS publishes again (docs/admin/dns.md), a few tens of seconds. Looked at 2026-11-09, knot 3.6: server.listen changes need a restart (no exposed listener opened on a reload), plain queries can't be refused by an ACL (query is for TSIG), and keeping its storage (a hostdisk volume) isn't enough alone: zonefile-load: difference-no-serial makes the zone file authoritative at start, ExternalDNS's records dropped again. Left: a storage kept with the zone file synced (zonefile-sync) or loaded from the journal, a design choice.
  • The embedded container images, faster still (ethere-container-images, 56 s at boot on 2 vCPU, Cilium's alone about 30 s): the kubelet started once the images it needs first are in, or a containerd store prebuilt with the image, if its snapshots work in the build's user namespace.
  • The control plane's rollout, faster still (playbooks/capi/controlplane.yml). Measured 2026-11-05 on the test cloud, 4 vCPU per server: 25 min per server, 76 min for three (39 min and 116 min before the idle gate, capi_prepare_ttl, one writer move and kubectl exec), against 14 to 17 min for replace.yml, measured on an older host. 8 vCPU: 73 min, the CPU no longer limits. 2 vCPU and 16 GB (OVH's r3-16): 102 min, 31 to 36 min per server. What is left at 4 vCPU, 24 runs of capi/controlplane.yml:
    • Between two runs: 26 min, the largest item (72 s each: the period of a minute and the wrapper's start), though the runs waiting for a server to boot would gain nothing; a watch on the Machines or a shorter period while a change is under way, and chaining in one run the steps that don't depend on a wait (one step per run today). Tried 2026-11-08: operator_sdk.util.requeue_after (10 s) after each step that the next one follows without waiting (a claim, a hook done, a Machine deleted, the git names), its signal a file the nested run writes and _runner.yml reads (ansible-operator runs nothing after the requeue): the next run came 18 to 23 s after instead of 69 s, but the rollout took 75 min alone and 81 at three jobs against 76: nothing measurable, removed. The runs themselves (40 to 60 s waiting, 4 to 5 min acting) are the larger part. Measured again 2026-11-10 (full tier): 84 min at 4 vCPU on the test cloud, 66 min on OVHcloud's r3-16 (102 before the fixes of 2026-11-08); the runs themselves are the larger part.
    • A run that only waits: 31 s, nearly all of it the nested ansible-playbook's start and its short tasks (14 min over the rollout).
    • git/writer.yml (about 90 s a run, 6 min), etcd/members.yml (about 60 s, 5 min), the server's stop (about 100 s, 3.5 min): its services' stop to look at, k8s/gateway.yml (about 55 s, 3 min).
    • The operator's eviction at each drain: about 1 min before it runs again, all its resources at once; keep it off the Machines about to go (the newest server preferred).
  • registry/push.yml right after a server's replacement on the qemu provider: its port-forward failed for over a minute in two runs (Zot ready), then held: restarted until it does, its cause not seen (the log goes with the run).
  • The external mode, not yet tested (docs/dev/cluster-api.md): the operator reaching OpenStack from the pods' network (the test cloud's catalog gives literal IPv4 addresses; a real cloud's names go through DNS64), a real api.<zone> through the delegation (the test answers it with a fake DNS server).
  • capi.test.js's two refused writes during a control plane rollout: two EOF 31 s apart, near the rollout's end, in every run. The test's dynamicTunnel going with its server's edge would give that, not proven: an outage through every edge is not ruled out. To tell them apart: write through two edges at once.
  • The Cinder CSI addon on a standalone cluster: its pods wait for a Secret that never comes (cinder-csi-cloud-config), harmless; leave the addon out of ethere-k8s-standalone at its next build.
  • On-demand GPU nodes for batch jobs (design in docs/dev/gpu.md): jobs declare their GPU requirements (node affinity on GPU Feature Discovery labels, DRA later), the operator picks the cheapest available flavor satisfying them from a flavor catalog, billing-aligned scale-down (delete idle nodes just before the end of each billed hour), safeguards; ethere-k8s-node-gpu image (NVIDIA open kernel modules only: Turing and newer GPUs, container toolkit, device plugin).
  • Provider independence (see docs/admin/providers.md): parametrize what is OVH-specific (vRack provider_network_type, Ext-Net, flavors, S3, port security) into a provider profile; run the provisioning on the test cloud with a vanilla profile; test the compatibility checklist (anycast/port security first) on two European candidates (Infomaniak, an SCS cloud), later one outside Europe.
  • Remove the Barbican leftovers (secret_store, openstack_secret lookup and action, test.sh).

Code

  • tests/k8s/kubeconfig.test.js's "the API by its public name once it resolves" fails (seen 2026-11-10, before and after the build workspace's move): k8s/kubeconfig.yml still runs "Get the servers of the cluster's region" (openstack.cloud.resources) with the tests' offline OpenStack environment, although k8s_api_public_name (localhost) resolves.

  • profile.d/20awscli.sh's __ethere_callback_aws reads a profile's S3 endpoint in the nested form (s3 =\n endpoint_url), which boto3 ignores: cluster configurations now declare it in a services section (dampetra's aws/config, 2026-10-26). Harmless while run-callbacks is a no-op; read the services form when it becomes one.

  • Translate the remaining French task names, comments and messages to English (as code is touched).