Exposing applications: the Gateway¶
Applications are exposed over HTTPS through one shared Gateway API Gateway, Cilium's, behind the edges. No
Octavia, no cloud load balancer: the edges' public addresses are published in DNS (short TTL), and each edge
forwards to the Gateway on its own server.
client ──TLS──▶ edge's HAProxy (:443, SNI) ──PROXY v2──▶ Cilium's Envoy (:8443, host network) ──▶ pods
│ ethere's names (git, the API): passed through as before
The path of a request¶
- The edge (
inventory/group_vars/edge/services.yml): HAProxy routes on the SNI without terminating TLS. The names it knows (the git replicas', the Kubernetes API's) go where they always went; any other name goes to the Gateway on the same server,[::1]:k8s_gateway_https_port(8443: Envoy listens on IPv6 only, the cluster has no IPv4), with a PROXY v2 header carrying the client's address. Port 80 (edge_http_port, opened in the edge's security group) goes tok8s_gateway_http_port(8080) the same way. - The Gateway (
k8s_cilium_values.gatewayAPI): Cilium's Envoy, in the agent (no Envoy DaemonSet), listening on the host's network of the nodes labelledethe.re/edge: "true"(theethere-k8s-aioservers: their edge is local), PROXY protocol required (enableProxyProtocol: only the edges reach it). It terminates TLS and routes to the applications' Services (X-Forwarded-For: the client's address, from the PROXY header). - The CRDs: Gateway API's standard channel (
k8s_addons_crds), pinned by sha256, applied by the control plane before the addons (ethere-addons.service); the GatewayClassciliumcomes with Cilium's chart.
The Gateway public (playbooks/k8s/gateway.yml)¶
One Gateway for the cluster, k8s_gateway_name (public) in the namespace k8s_gateway_namespace
(ethere-gateway), the routes' parent (parentRefs: [{name: public, namespace: ethere-gateway}]), applied from the workstation
(the names are the cluster's, the images every cluster's); provisioning/cluster.yml runs it, and
provisioning/replace.yml when an edge is replaced:
- one HTTPS listener per name of
k8s_gateway_hostnames(by default the cluster's zone and*.<the cluster's zone>, the first ofdns_zones), namedhttps-<name>(*.example.com→https-wildcard-example-com), its certificate the Secrettls-<name>in the Gateway's namespace, the applications' routes attaching from any namespace; - an
httplistener redirecting every request to HTTPS (301).
Its rename (2026-11)¶
It was ethere in the namespace gateway. k8s/gateway.yml deletes that namespace (temporary:
k8s_gateway_legacy_namespaces), whose Gateway would hold the ports, and makes the new one; its certificates are issued
again. The routes still naming the old Gateway answer nothing until their parentRefs name the new one: the tenants
change them.
Certificates: cert-manager (inventory/group_vars/k8s/cert-manager.yml)¶
An addon like Cilium (its chart rendered into the control plane's image, its images embedded there, on the control
plane only), in the namespace cert-manager (out of the default policies). With k8s_acme_server set, the
playbook creates the ClusterIssuer ethere-acme and annotates the Gateway with it: cert-manager's gateway-shim
makes a Certificate per HTTPS listener, into its Secret tls-<name>.
- DNS-01 through the hidden primary: dynamic updates (RFC 2136) to the Service
dns-primarywith the cluster's TSIG key (the Secretdns-tsig: cert-manager'sclusterResourceNamespaceis the primary's namespace,dns). Wildcards need DNS-01 anyway, and nothing has to reach the cluster on port 80 for a challenge. - A name outside the cluster's zones (
k8s.pmfor dampetra) has its_acme-challengeas a CNAME into the cluster's zone (_acme-challenge.k8s.pm→k8s.pm.acme.<zone>): cert-manager follows it (cnameStrategy: Follow) and writes the challenge there. The provider zone never holds a challenge, nor needs a credential of cert-manager's. k8s_acme_server(e.g. Let's Encrypt'shttps://acme-v02.api.letsencrypt.org/directory),k8s_acme_email,k8s_acme_ca_bundle(a private ACME server's CA: the test cloud's Pebble). Empty server: no issuer, the Secrets are made by hand.
Names¶
- The edges' name
k8s_gateway_edge_name(the cluster's zone itself: its apex): the public addresses of the edges having a Gateway (the running servers of the groupsedgeandk8s_node), aDNSEndpointExternalDNS publishes into the hidden primary. Load balancing by DNS, its TTL short (dns_ttl). A replacement takes the old server out first and waits for the caches (replace_edge_wait). - The routes' names inside the cluster's zones: ExternalDNS's source
gateway-httproutepublishes each accepted route's hostnames as a CNAME to the edges' name (the Gateway's annotationexternal-dns.kubernetes.io/target). Nothing to do for an application. A route on the edges' name itself (the documentation, at the zone's apex: no CNAME there) is left alone (external-dns.kubernetes.io/controller: none): its records are the edges' addresses. - Names in another zone (a provider zone of
dns_provider_zones, its adapter and credential): the playbook writes the zone's apex as the edges' addresses (no CNAME at an apex), any other name (a wildcard included) as a CNAME to the edges' name, and the challenges' CNAMEs (filterdns_gateway_records). A name in neither is refused. Records of names removed fromk8s_gateway_hostnamesstay: remove them by hand.
A cluster giving an application its own zone (dampetra):
k8s_gateway_hostnames: ["*.cloud.dampetra.com", k8s.pm, "*.k8s.pm"]
dns_provider_zones:
- {name: k8s.pm, provider: ovh, credentials: "{{ cluster_root_dir }}/secrets/ovh-api-k8s-pm.json"}
Exposing an application¶
For the applications: docs/exposing.md, served by the cluster (docs/admin/documentation.md). The Gateway's Envoy
reaches the pods as Cilium's ingress identity: an application lets it in (fromEntities: [ingress]), or the route
answers but the backend doesn't. The documentation's name (k8s_docs_hostname, the zone itself)
is routed by k8s/docs.yml.
k8s/configure.yml runs k8s/gateway.yml with the rest of the cluster's configuration; the operator's runs
(operator/reconcile.yml) leave the edges' name alone (k8s_gateway_edges: false): only the servers' replacements
change it, without a server about to go or still booting (edge_exclude).
Limits¶
- Dedicated
ethere-edgeservers have no Envoy of their own (no kubelet): an edge forwards to its own server's Gateway only. They will forward to the edge nodes (TODO.md). - One port per protocol on the Gateway: every HTTPS listener shares 8443 (distinguished by their hostnames), as Gateway API requires of listeners on one port.
- TLS is terminated in the cluster: the edges hold no key.
- The edges' addresses come from Nova when the playbook runs: an edge failing stays in the name until the next run (the operator will keep it up to date).
- The test cloud checks the whole path with Pebble (its validation skipped: the test zone isn't delegated) and
cert-manager's self-check pointed at the hidden primary (
tests/cluster/k8s-aio.test.js); in production the self-check asks the zone's public name servers.