Skip to content

Exposing applications: the Gateway

Applications are exposed over HTTPS through one shared Gateway API Gateway, Cilium's, behind the edges. No Octavia, no cloud load balancer: the edges' public addresses are published in DNS (short TTL), and each edge forwards to the Gateway on its own server.

client ──TLS──▶ edge's HAProxy (:443, SNI) ──PROXY v2──▶ Cilium's Envoy (:8443, host network) ──▶ pods
                    │ ethere's names (git, the API): passed through as before

The path of a request

  • The edge (inventory/group_vars/edge/services.yml): HAProxy routes on the SNI without terminating TLS. The names it knows (the git replicas', the Kubernetes API's) go where they always went; any other name goes to the Gateway on the same server, [::1]:k8s_gateway_https_port (8443: Envoy listens on IPv6 only, the cluster has no IPv4), with a PROXY v2 header carrying the client's address. Port 80 (edge_http_port, opened in the edge's security group) goes to k8s_gateway_http_port (8080) the same way.
  • The Gateway (k8s_cilium_values.gatewayAPI): Cilium's Envoy, in the agent (no Envoy DaemonSet), listening on the host's network of the nodes labelled ethe.re/edge: "true" (the ethere-k8s-aio servers: their edge is local), PROXY protocol required (enableProxyProtocol: only the edges reach it). It terminates TLS and routes to the applications' Services (X-Forwarded-For: the client's address, from the PROXY header).
  • The CRDs: Gateway API's standard channel (k8s_addons_crds), pinned by sha256, applied by the control plane before the addons (ethere-addons.service); the GatewayClass cilium comes with Cilium's chart.

The Gateway public (playbooks/k8s/gateway.yml)

One Gateway for the cluster, k8s_gateway_name (public) in the namespace k8s_gateway_namespace (ethere-gateway), the routes' parent (parentRefs: [{name: public, namespace: ethere-gateway}]), applied from the workstation (the names are the cluster's, the images every cluster's); provisioning/cluster.yml runs it, and provisioning/replace.yml when an edge is replaced:

  • one HTTPS listener per name of k8s_gateway_hostnames (by default the cluster's zone and *.<the cluster's zone>, the first of dns_zones), named https-<name> (*.example.com → https-wildcard-example-com), its certificate the Secret tls-<name> in the Gateway's namespace, the applications' routes attaching from any namespace;
  • an http listener redirecting every request to HTTPS (301).
ansible-playbook $ETHERE_ROOT_DIR/playbooks/k8s/gateway.yml

Its rename (2026-11)

It was ethere in the namespace gateway. k8s/gateway.yml deletes that namespace (temporary: k8s_gateway_legacy_namespaces), whose Gateway would hold the ports, and makes the new one; its certificates are issued again. The routes still naming the old Gateway answer nothing until their parentRefs name the new one: the tenants change them.

Certificates: cert-manager (inventory/group_vars/k8s/cert-manager.yml)

An addon like Cilium (its chart rendered into the control plane's image, its images embedded there, on the control plane only), in the namespace cert-manager (out of the default policies). With k8s_acme_server set, the playbook creates the ClusterIssuer ethere-acme and annotates the Gateway with it: cert-manager's gateway-shim makes a Certificate per HTTPS listener, into its Secret tls-<name>.

  • DNS-01 through the hidden primary: dynamic updates (RFC 2136) to the Service dns-primary with the cluster's TSIG key (the Secret dns-tsig: cert-manager's clusterResourceNamespace is the primary's namespace, dns). Wildcards need DNS-01 anyway, and nothing has to reach the cluster on port 80 for a challenge.
  • A name outside the cluster's zones (k8s.pm for dampetra) has its _acme-challenge as a CNAME into the cluster's zone (_acme-challenge.k8s.pm → k8s.pm.acme.<zone>): cert-manager follows it (cnameStrategy: Follow) and writes the challenge there. The provider zone never holds a challenge, nor needs a credential of cert-manager's.
  • k8s_acme_server (e.g. Let's Encrypt's https://acme-v02.api.letsencrypt.org/directory), k8s_acme_email, k8s_acme_ca_bundle (a private ACME server's CA: the test cloud's Pebble). Empty server: no issuer, the Secrets are made by hand.

Names

  • The edges' name k8s_gateway_edge_name (the cluster's zone itself: its apex): the public addresses of the edges having a Gateway (the running servers of the groups edge and k8s_node), a DNSEndpoint ExternalDNS publishes into the hidden primary. Load balancing by DNS, its TTL short (dns_ttl). A replacement takes the old server out first and waits for the caches (replace_edge_wait).
  • The routes' names inside the cluster's zones: ExternalDNS's source gateway-httproute publishes each accepted route's hostnames as a CNAME to the edges' name (the Gateway's annotation external-dns.kubernetes.io/target). Nothing to do for an application. A route on the edges' name itself (the documentation, at the zone's apex: no CNAME there) is left alone (external-dns.kubernetes.io/controller: none): its records are the edges' addresses.
  • Names in another zone (a provider zone of dns_provider_zones, its adapter and credential): the playbook writes the zone's apex as the edges' addresses (no CNAME at an apex), any other name (a wildcard included) as a CNAME to the edges' name, and the challenges' CNAMEs (filter dns_gateway_records). A name in neither is refused. Records of names removed from k8s_gateway_hostnames stay: remove them by hand.

A cluster giving an application its own zone (dampetra):

k8s_gateway_hostnames: ["*.cloud.dampetra.com", k8s.pm, "*.k8s.pm"]
dns_provider_zones:
  - {name: k8s.pm, provider: ovh, credentials: "{{ cluster_root_dir }}/secrets/ovh-api-k8s-pm.json"}

Exposing an application

For the applications: docs/exposing.md, served by the cluster (docs/admin/documentation.md). The Gateway's Envoy reaches the pods as Cilium's ingress identity: an application lets it in (fromEntities: [ingress]), or the route answers but the backend doesn't. The documentation's name (k8s_docs_hostname, the zone itself) is routed by k8s/docs.yml.

k8s/configure.yml runs k8s/gateway.yml with the rest of the cluster's configuration; the operator's runs (operator/reconcile.yml) leave the edges' name alone (k8s_gateway_edges: false): only the servers' replacements change it, without a server about to go or still booting (edge_exclude).

Limits

  • Dedicated ethere-edge servers have no Envoy of their own (no kubelet): an edge forwards to its own server's Gateway only. They will forward to the edge nodes (TODO.md).
  • One port per protocol on the Gateway: every HTTPS listener shares 8443 (distinguished by their hostnames), as Gateway API requires of listeners on one port.
  • TLS is terminated in the cluster: the edges hold no key.
  • The edges' addresses come from Nova when the playbook runs: an edge failing stays in the name until the next run (the operator will keep it up to date).
  • The test cloud checks the whole path with Pebble (its validation skipped: the test zone isn't delegated) and cert-manager's self-check pointed at the hidden primary (tests/cluster/k8s-aio.test.js); in production the self-check asks the zone's public name servers.