Skip to content

Container registry

The cluster's own registry: the images built for it (the operator's first, TODO.md) and a pull-through cache of the public registries for every pod. Zot (OCI-native, one binary), its content in S3: the servers running it stay cattle, a rotation loses nothing. Decided 2026-11-01; tested on the test cloud (tests/cluster/registry.test.js).

What doesn't depend on it

Whatever a server needs to boot and join is embedded in its image, as before (docs/admin/volumes.md, the addons): Cilium, CoreDNS, the sandbox image, the addons' images, Zot's own. The registry serves the workloads only, which need Cilium anyway: it is reached through a Kubernetes Service, not through an anycast network of its own like etcd, the API or git (services used outside Kubernetes or before it).

The service

  • Zot as a DaemonSet on the control plane servers (inventory/group_vars/k8s/registry.yml, an addon: rendered into the control plane's image, applied at boot), on the host's network like the Cinder CSI controller: the pods are IPv6-only without 464XLAT (docs/admin/egress.md), and an S3 endpoint or an upstream registry may be IPv4 only. It listens on every address (k8s_registry_host_port, 5443): the Service through the node's, a port-forward through localhost; the image's security group lets nothing reach it from outside. It resolves names with the node's resolvers (dnsPolicy: Default): CoreDNS's DNS64 answers lead to the NAT64 prefix, which only the pods reach. In the namespace ethere-registry (ethere's own namespaces are prefixed ethere-), exempt from the default network policies like the other system namespaces (docs/admin/network-policies.md).
  • One Service, its cluster IP fixed (k8s_registry_ip, in k8s_services_cidr): containerd reaches it through Cilium's socket load balancing on every node; an instance failing its readiness probe (/readyz, outside the authentication) leaves the Service. sessionAffinity: ClientIP: a client stays on one instance.
  • The instances share nothing but the bucket: each one reads and writes S3 directly, without deduplication (its local database can't be shared) nor Zot's scale-out clustering (members listed in advance, repositories sharded between them: neither ethere's way nor availability). No garbage collection yet: several instances collecting the same bucket could delete an upload in progress on another one (TODO.md).
  • Its configuration from the playbook playbooks/k8s/registry.yml, not from the image (an image serves every cloud): the ConfigMap zot-config (S3 endpoint, region, bucket, the pushers) and the Secret zot-s3 (the S3 user's keys, as environment variables: the storage driver's default credentials). The pods wait for them, like the Cinder CSI driver for its credentials, and restart when they change. The bucket and its S3 user are made as for etcd's backups (etcd/backup.yml: ovh_s3_user, its keys only in the cluster's Secret), by default with the same settings (k8s_registry_s3_endpoint, _region and _user default to etcd_backup_s3_*). k8s/configure.yml runs it.
  • Long requests and responses: Zot cuts both after 60 s by default (http.readTimeout, writeTimeout), which a large layer streamed from S3 or pushed exceeds (a 1 GB layer: containerd fails with tls: bad record MAC, the node never gets it). Both are k8s_registry_http_timeout (30 min).

TLS and who may do what

  • Its server certificate is enrolled (docs/admin/pki.md) by the control plane servers (registry, CA registry), its files alone mounted by hostPath (not the server's other keys): the cluster IP, the Service's names, k8s_registry_name (registry.ethere.internal, the name images are referenced by) and localhost.
  • Nothing anonymous: every client presents a certificate of the CA registry (Zot takes the user from its CN; a client without one would be anonymous, and anonymous clients get nothing): inside the cluster, a pod of a system namespace, a host network pod or an application allowed to reach the Service would read every image of the cluster, and have the cache fetch any public image; and the registry exposed one day (TODO.md), its images would be public.
  • The nodes read with a client certificate they enroll (registry-client), the same CN for all (ethere-node: the servers' names change with every rotation, Zot only knows the CN); containerd presents it and checks the registry's certificate against its CA bundle (which comes with it, docs/admin/pki.md).
  • The pushers read and write: the CNs the playbook lists (the admins of ethere.yaml as their certificates name them, ethere-admin-<name>, and k8s_registry_extra_pushers: later the operator). A pusher's certificate comes from the CA registry, signed on the workstation like the git signer's (utils/_signer_certificate.yml), short-lived, kept in tmpfs. A pod pulling images itself (a CI) will get its own.

containerd

config_path (/etc/containerd/certs.d, baked into the images running a kubelet, k8s_node/services.yml):

  • k8s_registry_name: the Service's cluster IP, the CA bundle and the client certificate;
  • each public registry cached (k8s_registry_mirrors: docker.io, ghcr.io, quay.io, registry.k8s.io): Zot first, under the registry's prefix (override_path), then the registry itself, which containerd falls back to: a registry down degrades to direct pulls.

Zot's sync extension fetches a missing image on demand (onDemand) from its registry and keeps it under that registry's name (docker.io/library/nginx), so the names of different registries never collide; it assembles it in a local directory first (downloadDir, required with a remote storage). The sources are the playbook's (k8s_registry_mirrors, the configuration), the hosts containerd sends to Zot the image's: a cluster changing the list rebuilds its images. In the bucket, Zot adds its own prefix under the driver's root (zot/zot/<repository>/).

Pushing

From the workstation, through the Kubernetes API (kubectl port-forward to the Service, through the edge as for any admin access), with regctl (regclient: client certificates, which crane lacks; installed by the profile):

ansible-playbook $ETHERE_ROOT_DIR/playbooks/registry/push.yml -e registry_push_source=ocidir://<dir>:<tag> -e registry_push_repository=<name:tag>

The source as regctl names it (an OCI layout, another registry's image); the image is then registry.ethere.internal/<name:tag> for the cluster's pods. The certificate is one-time (two hours), its key and regctl's configuration in a private directory of $XDG_RUNTIME_DIR, removed at the end.

The public registry

The members push through registry.<zone> (k8s/registry_public.yml, part of k8s/configure.yml when the cluster has identities): a Zot of its own, a Deployment (zot-public, two replicas on the pods' network) on the same bucket and the same storage settings (read from the ConfigMap zot-config), so an image pushed there is the nodes' at once. Behind the Gateway in plain HTTP (TLS at the Gateway: its certificate for the zone), its route's timeouts lifted like Zot's own (k8s_registry_http_timeout).

  • Signing in: OpenID Connect through Dex (its client zot, a secret k8s/identity.yml generates in the cluster: the Secret dex-zot), then API keys (zak_…) for docker login (the email, the key). Dex's network policy admits the zot-public pods: they reach it by its public name, through the edge, which passes their address on.
  • Who may do what: k8s_registry_public_repositories, a pattern per group list ({"aubonticket/**": [aubonticket]}: every action), k8s_registry_public_admin_groups for everything; nothing anonymous, nothing for anyone else.
  • Its database (users, API keys, sessions' groups) in Valkey, shared by the replicas: the valkey chart (k8s_valkey_*), a primary, replicas and three Sentinels on Cinder volumes, its passwords generated in the cluster (the Secret valkey-users); Zot asks Sentinel for the primary (as its user sentinel: the chart turns default off there). Its session keys too (zot-public-keys), so a session holds on every replica.

Why not the DaemonSet itself: Zot's database on Redis makes Redis necessary to every request (a pull answers 500 with Redis down, even with a client certificate) and to its start (it exits). The nodes' registry keeps its local database: Valkey down, the nodes pull; only the members' sign-ins and pushes wait.