Skip to content

The pods' way out

The clusters are IPv6-only: the pods have private addresses (ipv6_private_prefixes.k8s_pods); the servers have a public IPv4 address and, where the provider gives one, a public IPv6 address (OVHcloud: Ext-Net and Ext-Net-v6). The pods reach the outside transparently, every protocol, without any configuration of their own:

  • toward an IPv6 address: directly, masqueraded by Cilium behind their node's public IPv6 address;
  • toward an IPv4-only name: through NAT64 on their node (DNS64 gives the name an address in 64:ff9b::/96), out behind their node's IPv4 address.

The cluster stays IPv6-only (no IPv4 on the pods, no dual stack), and every node is its own way out: no gateway server, no anycast hop.

What it means for an application: docs/network.md, served by the cluster (docs/admin/documentation.md).

How it works

  • DNS (inventory/group_vars/k8s/addons.yml, k8s_coredns_*): CoreDNS resolves the cluster's names and forwards the others to its node's resolvers, those systemd-resolved got from DHCP, as the pods reach them: an IPv4 one followed by its address in the NAT64 prefix (ethere-pods-resolv, k8s_node/services.yml: the kubelet's resolvConf, /run/ethere/pods-resolv.conf, written again when systemd-resolved's changes; the host network pods use the IPv4 one, NAT64 translating the pods' addresses only). Its dns64 plugin synthesizes an AAAA record in 64:ff9b::/96 for a name having only an A record; a name having an AAAA record keeps it (not translate_all).
  • NAT64 (inventory/group_vars/k8s/nat64.yml): kubernetes-sigs/nat64, a DaemonSet on every node (its image embedded like the other addons'). Its agent adds a dummy interface nat64 holding k8s_nat64_prefix and k8s_nat64_ipv4_cidr; an eBPF program there translates a pod's packet to IPv4 (its source the pod address' last 16 bits in 198.18.0.0/16), nftables masquerades it behind the node's IPv4 address, and the replies take the way back. Hence k8s_pods_node_mask: 112 (a node's pod prefix must fit the IPv4 range) and CONFIG_DUMMY in the kernel. An init container adds a last-resort route to the prefix: the agent requires one when it starts.
  • IPv6 (k8s_cilium_values): Cilium's BPF masquerade, its ip-masq-agent leaving out the private prefixes (pods and servers: the cluster's own traffic keeps its addresses) and the NAT64 prefix (translated from the pods' own addresses). The node's IPv6 default route comes from its external network's gateway (the region profile, servers_external_networks). Cilium never attaches to the agent's interface (devices: "!nat64").

Limits

  • IPv4 literals: a program connecting to an IPv4 address written in its configuration (not a name) doesn't go out: there is no 464XLAT on the pods. Use a name, or 64:ff9b::<IPv4> by hand.
  • IPv6 destinations need the nodes' IPv6 route: without a gateway on an IPv6 external network (or without one at all), the names having an AAAA record are unreachable, even when they have an A record too (DNS64 keeps their AAAA).
  • No filtering: every pod reaches every destination; Cilium's network policies (toFQDNs, CIDRs) restrict it when needed.
  • The nodes' public addresses are the pods' source addresses: an allowlist at a third party lists every node.