The pods' way out¶
The clusters are IPv6-only: the pods have private addresses (ipv6_private_prefixes.k8s_pods); the servers have
a public IPv4 address and, where the provider gives one, a public IPv6 address (OVHcloud: Ext-Net and
Ext-Net-v6). The pods reach the outside transparently, every protocol, without any configuration of their own:
- toward an IPv6 address: directly, masqueraded by Cilium behind their node's public IPv6 address;
- toward an IPv4-only name: through NAT64 on their node (DNS64 gives the name an address in
64:ff9b::/96), out behind their node's IPv4 address.
The cluster stays IPv6-only (no IPv4 on the pods, no dual stack), and every node is its own way out: no gateway server, no anycast hop.
What it means for an application: docs/network.md, served by the cluster (docs/admin/documentation.md).
How it works¶
- DNS (
inventory/group_vars/k8s/addons.yml,k8s_coredns_*): CoreDNS resolves the cluster's names and forwards the others to its node's resolvers, those systemd-resolved got from DHCP, as the pods reach them: an IPv4 one followed by its address in the NAT64 prefix (ethere-pods-resolv,k8s_node/services.yml: the kubelet'sresolvConf,/run/ethere/pods-resolv.conf, written again when systemd-resolved's changes; the host network pods use the IPv4 one, NAT64 translating the pods' addresses only). Itsdns64plugin synthesizes an AAAA record in64:ff9b::/96for a name having only an A record; a name having an AAAA record keeps it (nottranslate_all). - NAT64 (
inventory/group_vars/k8s/nat64.yml): kubernetes-sigs/nat64, a DaemonSet on every node (its image embedded like the other addons'). Its agent adds a dummy interfacenat64holdingk8s_nat64_prefixandk8s_nat64_ipv4_cidr; an eBPF program there translates a pod's packet to IPv4 (its source the pod address' last 16 bits in198.18.0.0/16), nftables masquerades it behind the node's IPv4 address, and the replies take the way back. Hencek8s_pods_node_mask: 112(a node's pod prefix must fit the IPv4 range) andCONFIG_DUMMYin the kernel. An init container adds a last-resort route to the prefix: the agent requires one when it starts. - IPv6 (
k8s_cilium_values): Cilium's BPF masquerade, its ip-masq-agent leaving out the private prefixes (pods and servers: the cluster's own traffic keeps its addresses) and the NAT64 prefix (translated from the pods' own addresses). The node's IPv6 default route comes from its external network'sgateway(the region profile,servers_external_networks). Cilium never attaches to the agent's interface (devices: "!nat64").
Limits¶
- IPv4 literals: a program connecting to an IPv4 address written in its configuration (not a name) doesn't go
out: there is no 464XLAT on the pods. Use a name, or
64:ff9b::<IPv4>by hand. - IPv6 destinations need the nodes' IPv6 route: without a
gatewayon an IPv6 external network (or without one at all), the names having an AAAA record are unreachable, even when they have an A record too (DNS64 keeps their AAAA). - No filtering: every pod reaches every destination; Cilium's network policies (
toFQDNs, CIDRs) restrict it when needed. - The nodes' public addresses are the pods' source addresses: an allowlist at a third party lists every node.