DNS¶
DNS is a key component of the architecture: it does the load balancing of exposed services (short TTL, records updated by the operator, phased rotations waiting for the TTL), so its update latency directly bounds failover and rotation times. It also designates the writer of the replicated git repositories, in an internal zone the hidden primary alone serves (see git.md).
Roles¶
Three independent roles, which may or may not be held by the same company (for dampetra, OVHcloud holds all three, by coincidence):
- OpenStack provider: runs the cluster, the hidden primary and the secondaries (
providers.md). - Registrar: holds the domain's delegation (the NS records in the parent zone) and its DS records (DNSSEC).
- Host of the zone holding the NS names (their A/AAAA records, updated by the operator through an API): the registrar or any DNS provider.
ethere must never assume they are the same: each one has its own settings (and API credentials) in the cluster's configuration.
Who manages what¶
- An Ansible adapter per DNS provider, no OpenTofu (
provisioning/dns.yml, implemented 2026-10-09): the provider zones the cluster manages are listed indns_provider_zones({name, provider, credentials}: a sops-encrypted API credential scoped to that zone), each written through its provider's module (plugins/modules/<provider>_dns_records.py, only the standard library:ovh_dns_recordsfor OVHcloud), which makes RRsets match the wanted ones (check mode and diff: the plan). The playbook derives everything fromdns_zonesand the cluster's servers: each zone's name servers are the newestethere-dns-secondaryservers (as many as names), their addresses the servers' public ones; the delegation (NS records) goes into the provider zone that is the zone's parent; a name server's A/AAAA records into the provider zone holding its name, or, named inside the cluster's zone (ns1.<zone>), into that zone on the hidden primary (DNSEndpointobjectsname-server-<name>, written withkubernetes.core.k8sthrough the admin kubeconfig: runk8s/kubeconfig.ymlfirst; ExternalDNS publishes them, so they survive the primary's moves) and as glue into the parent. Written in that order, the NS records last, once every secondary serves the zone. Run again after a secondary is replaced. Having a usable API (scoped credentials) becomes a selection criterion for a DNS provider, like for identities inproviders.md. - The NS names' A/AAAA records: they change on every replacement of a secondary, with the TTL choreography below:
provisioning/dns.ymlsets them for now, the operator will. - DS records: knot rolls its keys automatically, and the DS at the registrar must follow each KSK rollover. Preferably through CDS/CDNSKEY (RFC 7344/8078) when the registry supports it: it reads the new key published in the zone, nobody has anything to do. Otherwise the operator updates the DS through the registrar's API (the provisioning only sets the initial one, if at all).
Implementation (first increment, 2026-10-08)¶
On the test cloud only (no registrar, no provider zone yet): inventory/group_vars/k8s/dns.yml (primary), inventory/group_vars/dns_secondary/ (the ethere-dns-secondary image), inventory/group_vars/all/dns.yml (dns_zones), tests/cluster/dns.test.js.
- DNS-Net: a private network for the transfers, with an anycast address. The control plane servers and the secondaries have a port there.
- The hidden primary runs on the control plane: knot in a pod (one replica,
Recreate, the host's network,nodeSelector ethe.re/control-plane, the control-plane taint tolerated), which binds DNS-Net's anycast address while it runs (an init container binds it, a sidecar removes it): the secondaries only know that address, and the primary may move between control plane servers. Its image is embedded into the control plane's image, its manifests are part of the addons. - Transfers over QUIC with mutual TLS (XoQ), not TSIG: enrolled certificates (CA
dns), checked by name on both ends (cert-hostname:dns-primary,dns-secondary); no shared secret on the secondaries. Checked in a local spike: certificates of an unknown name or another CA are refused. - No NOTIFY: the primary doesn't know the secondaries (they'd have to be listed); they check its SOA every
dns_soa_refreshseconds (5 s: an update reaches them within 0.7–5 s in the spike). NOTIFY may come later, from the NS records (a sidecar resolving them) or from the secondaries' own requests. - Dynamic updates from the cluster with TSIG (public zones): the key is the Secret
dns-tsig, created by the first control plane server (ethere-addons.service), replaced byplaybooks/dns/tsig.yml(a new key, the primary restarted on it and checked to run with it); the Servicedns-primaryis the cluster's way to the primary. - The records come from Kubernetes through ExternalDNS (redesign P5,
inventory/group_vars/k8s/external-dns.yml): ExternalDNS v0.24.0 as an addon (its image embedded in the control plane's, its CRDDNSEndpointvendored inshare/k8s/), like the primary one replica on the host's network of a control plane server, reconciles theDNSEndpointobjects into the primary (--events: a change is served within seconds;--registry txt --policy sync: what it doesn't own is left alone;--rfc2136-batch-change-size 1: a refused record doesn't block the others). It speaks DNS over TLS on port 853 (TCP; QUIC transfers use 853/UDP) of DNS-Net's anycast address, at the primary's namedns_primary_fqdn(mapped by the node's hosts file; a SAN of thedns-primarycertificate, which ExternalDNS checks), authenticated by its enrolled client certificateexternal-dns(CAdns), no TSIG. The primary's ACLexternal-dnsallows it updates of A, AAAA and TXT and zone transfers, nothing else. Its zones (the public ones and the internal one) come from the ConfigMapexternal-dns-zones, written by the control plane. Nothing outside the cluster updates the primary: the workstation writesDNSEndpointobjects (provisioning/dns.yml,git/writer.yml) through the Kubernetes API. - The internal zone
ethere.internal(dns_internal_zone): the cluster's internal names (the git replicas' and writer's,git.md), unsigned, updated by ExternalDNS alone, served by the primary alone on DNS-Net and never transferred (the secondaries only ask for the zones of their user-data). It is baked into the control plane's image (k8s_ctrl/dns.yml), apart fromdns_zones, and put into the ConfigMapdns-primary-zones. - State: none on the primary: its storage is an
emptyDir, its serial a Unix time (serial-policy: unixtime: the secondaries accept the zone again after a move), its records published again by ExternalDNS. The primary and ExternalDNS tolerate an unreachable or not-ready node for 30 s, then move. Only its DNSSEC keys are kept: exported (a consistentkeymgrsnapshot, every 30 s) through a smallhostPath(k8s_dns_storage_dir) and put into the Secretdns-primary-keysby the server running it, imported bykeymgr import-pemwhen it starts on another server: the DNSKEYs (hence the DS) don't change (checked in the spike). - No transfer before its records are back: a primary just started holds the zones' base alone, its serial above the secondaries'; it refuses their transfers until ExternalDNS has published its records again (its ownership TXT records in every public zone, then a few seconds), or after
k8s_dns_transfers_timeout, and is Ready only then (k8s/dns.yml: thetransferssidecar). Meanwhile the secondaries keep the zone they have. Before, a primary moved by a drain made them answer NXDOMAIN for every name of ExternalDNS for 30 s (OVHcloud, 2026-11-08). A standalone cluster's primary answers the public itself: it still does there. The secondaries retry a zone never transferred everydns_soa_refreshseconds (retry-max-interval), not knot's backoff (up to two hours), which left a secondary created before the control plane unserved long after the primary answered (pki.test.js, failing 3 runs of 4). - The zones are deployment data (
dns_zones): the user-data of the control plane servers and the secondaries writes their zone files; the control plane gives them to the primary (ConfigMapdns-primary-zones, created by the first control plane server only: the user-data is never updated, and a server rebooting would otherwise undo a later change), the secondaries serve the zones found there. A zone's base changes afterwards (a name server added) throughprovisioning/dns.yml: it patches the ConfigMap withdns_zone_filesand restarts the primary, which is stateless (ExternalDNS publishes its records again within a minute or two; meanwhile the zone holds its base alone). - The secondaries answer on every address: resolved's stub listener is turned off there (it holds port 53;
/etc/resolv.confpoints at resolved's upstream servers instead,images_resolv_conf), and the image's security group opens port 53 (UDP and TCP) on their external ports (servers_security_group_rules__dns_secondary). - knot 3.6 is packaged in ethere's repository (QUIC through ngtcp2 with GnuTLS) for the secondaries; the primary runs the upstream image.
Hidden primary (design)¶
- Primary: an authoritative server (knot) inside the cluster, not exposed publicly. The operator updates records with dynamic updates (RFC 2136) authenticated by TSIG. knot signs the zone (DNSSEC, automatic key management).
- Public secondaries: our own knot instances on small instances (e.g. OVH
d2-2, ~€6/month), one per availability zone of the cluster's 3-AZ region (if the region fails, the services are down anyway; disaster recovery rebuilds everything in another 3-AZ region), each with its own public IPv4 and IPv6: they serve the zone to the Internet, kept in sync by NOTIFY then incremental transfers (IXFR), authenticated by TSIG. They are ethere instances like the others: adns-secondaryimage/role, cattle, with no post-boot configuration (they only need the primary's address and the TSIG key). - Transfers over the private network: the OVH vRack spans regions, so NOTIFY and AXFR/IXFR go through it and the primary is never exposed. A secondary outside OVH (if ever needed) would reach the cluster through WireGuard.
- NS names outside the zone, in a provider-managed zone (e.g.
ns1.dns.<another domain>in an OVH DNS zone), with A and AAAA records and a short TTL: no glue at the registrar. The delegation (NS names) never changes; only these records are delegated to the provider. - Replacing a secondary follows the same phased choreography as the services: create the new instance, update the A/AAAA of its NS name through the provider's API, wait for the TTL, delete the old instance. Resolvers still asking the old address fall back to the other NS anyway. A disaster recovery to another region is the same operation for the three names.
- No floating IPs: they would only avoid these record updates, at the cost of a Public Cloud Gateway in front of all public DNS traffic (throughput cap, possible single point of failure), no IPv6, and extra cost. Their only real use would be in-zone NS names with glue, rejected for disaster recovery (glue is cached for days by the parent).
- Rough cost: 3 × d2-2 ≈ €17/month.
Records published by the cluster¶
Since the whole chain is ours, the cluster can publish its own records on the hidden primary, with standard tools speaking RFC 2136 (TSIG dynamic updates), like the operator:
- cert-manager, ACME DNS-01
rfc2136solver: Let's Encrypt certificates, wildcards included, without exposing port 80; propagation to the secondaries takes ~1 s (local POC). - ExternalDNS,
rfc2136provider: A/AAAA/CNAME/TXT records from Services, Ingresses or the Gateway API. - Internal names of the servers (instead of the user-data's
/etc/hosts).
Rules:
- One credential per client, scoped by knot ACLs (a TSIG key, or as implemented for ExternalDNS a client certificate matched by
cert-hostname) (update-type,update-owner,update-owner-match,update-owner-name): cert-manager only TXT under_acme-challenge.*; ExternalDNS only A/AAAA/CNAME/TXT under an application subdomain; the operator the load-balancing and infrastructure names. A leaked key can't rewrite the zone. - Don't bypass the DNS choreography: ExternalDNS updates immediately, without waiting for TTLs. Applications publish CNAMEs to load-balancing names managed by the operator (e.g.
app.example.com → lb.example.com); only the operator manages the A/AAAA of those names, with phased rotations. ExternalDNS's TXT ownership records keep it from touching the operator's records.
Why¶
- Control and speed of an internal DNS: no provider API, rate limits or zone "refresh"; an update reaches the secondaries in seconds (to be measured, see below).
- Resilience: the secondaries are in several regions and keep serving the zone if the cluster is lost (until the SOA expire); the primary is never exposed. Public exposure of small instances (100 Mbit/s) is mitigated by the provider's anti-DDoS and knot's response rate limiting, and resolvers fall back to the other NS.
- Disaster recovery: a cluster rebuilt in another region only needs to be allowed to transfer the zone (TSIG key), with no change at the registrar. NS/glue changes, cached for days at the parent, are never needed.
- No stable public IP is needed for name servers (no glue), which would otherwise conflict with cattle instances.
- Same software on both ends: pre-signed DNSSEC, no minimum TTL, NOTIFY propagation bounded by our own servers.
- Provider diversity isn't a goal while the whole platform runs on one provider: if all its regions fail, the services are down anyway. (Exceptions to keep in mind: a maintenance page hosted elsewhere, or a disaster recovery target on another cloud, which would need a secondary there.)
Rejected options¶
- Everything at the provider: update propagation through the provider's API can take minutes (OVH applies zone changes after a "refresh"), added to the TTL at each phase of a rotation; API rate limits, minimum TTLs, one driver per provider.
- Everything in-house, only NS/glue at the provider: the NS need stable public IPs (glue), incompatible with rotating instances; after a region loss, the glue points to dead addresses and takes hours or days to change; the authoritative servers are exposed (DDoS amplification).
Assumptions to check¶
- Dynamic updates with TSIG on knot, automatic DNSSEC signing, and NOTIFY + IXFR to a different implementation work together, with an update → served-by-the-secondary latency of a few seconds. → local POC (
poc/dns/). - The secondaries keep serving (signed) records when the primary is down. → local POC.
- Replacing a secondary: updating the A/AAAA of its NS name through the OVH API, and how resolvers behave until the TTL expires. → test on OVH.
- Transfers from the primary to the secondaries over the private network, update latency. → test on OVH.
Local POC results (poc/dns/run.sh, 2026-10-06)¶
knot 3.6.0 as the hidden primary, bind 9.20 as the secondary (rootless podman, host networking):
- Assumption 1 confirmed: TSIG dynamic updates, automatic DNSSEC signing (ECDSA P-256) and NOTIFY + IXFR between knot and bind work together. Update on the primary → served by the secondary: ~1.1 s (20 updates: min 1076 ms, median 1155 ms, max 1202 ms); same for a rotation-like replacement (delete + add). Transfers after the first stay small while the zone grows (incremental). Updates without the TSIG key are refused.
- Assumption 2 confirmed: with the primary down, the secondary keeps serving the signed records.
- Scoped TSIG keys confirmed (knot ACLs
update-type+update-owner: name): anacmekey limited to TXT records named_acme-challenge/_acme-challenge.<label>…(update-owner-match: pattern, one pattern per depth:_acme-challenge,_acme-challenge.*,_acme-challenge.*.*), anexternaldnskey limited to A/AAAA/CNAME/TXT strictly underapps.<zone>(update-owner-match: sub); everything outside is refused, the operator key writes anywhere, and the ACME record reaches the secondary signed. - Assumption 3 holds (2026-10-27, dampetra): both secondaries replaced one after the other, the name servers' new addresses published as glue at OVHcloud within seconds, the old secondaries deleted once the 300 s TTL passed, the zone served throughout (
provisioning/replace.ymldoes it now). Assumption 4 holds (2026-10-26, dampetra): the secondary in GRA9 transfers the zone from the primary in EU-WEST-PAR over DNS-Net, one vRack VLAN across the two regions; the update latency is not measured yet.
Commercial secondaries (option, not the target)¶
Kept for reference: adding a commercial anycast secondary later (e.g. RcodeZero) would strengthen the public serving.
OVH's secondary DNS (checked 2026-10-06 through the OVHcloud API, poc/ovh/api.js)¶
- OVH's secondary DNS is only available with a dedicated server or a VPS (
/dedicated/server/{s}/secondaryDnsDomains,/vps/{s}/secondaryDnsDomains); there is nothing equivalent for Public Cloud projects. Adding a zone takes the domain and an IPv4ip(the primary's), and the service is attached to the server: using it would require a VPS, and possibly the primary to be that VPS. - The account has no VPS nor dedicated server, so OVH's secondaries can't be tested as is.
- The design doesn't depend on OVH: any secondary DNS provider accepting NOTIFY + IXFR (ideally TSIG) from a primary of ours fits.
European secondary providers (compared 2026-10-06 from their public documentation)¶
Several providers will be combined. Since the hidden primary signs the zone, every secondary serves the same pre-signed zone: multi-provider DNSSEC needs no multi-signer setup, as long as each provider serves pre-signed zones as-is.
| Provider (country) | Anycast | NOTIFY | Transfers | TSIG | Pre-signed DNSSEC | Notes |
|---|---|---|---|---|---|---|
| RcodeZero, by nic.at (AT) | yes, 2 anycast clouds (distinct AS), 20+ sites | yes | AXFR | yes | yes (documented) | TLD-operator grade; paid per zone (Starter 5 zones, ~€3.98/extra zone/month) |
| Netnod DNSNODE (SE) | yes, 80+ nodes | yes | AXFR/IXFR, "seconds" | yes | to check | TLD/enterprise grade, price on request |
| ClouDNS (BG) | yes, 60+ POPs | to check | AXFR/IXFR | yes | to check | cheap plans (from ~$3/month) |
| Hetzner DNS (DE) | to check | yes | AXFR (full) | yes | to check | free (25 zones), minimum TTL 60 s for records |
| INWX (DE) | yes, 64 nodes, 3 AS | to check | to check | yes | to check | documented as primary allowing transfers; acting as secondary for an external primary to confirm |
| le_dns (FR/DE) | no (2 servers) | yes | AXFR/IXFR | yes | not stated | free (5 zones), single operator, no SLA: fine for a test, not for production |
| DNScale (EE) | yes | — | no zone transfers (API sync only) | — | — | excluded |
If ever needed: RcodeZero as the main commercial secondary (pre-signed DNSSEC confirmed, two independent anycast networks), plus a second provider (Netnod, ClouDNS or Hetzner) after checking the "to check" cells — notably pre-signed DNSSEC and the minimum TTL, which bounds the TTL of the load-balancing records.
Out of scope for now¶
- Internal name resolution of the servers (today a static
/etc/hostswritten by the user-data,servers_domain). The internal zoneethere.internalexists, for the git names only. - OpenStack Designate is not needed: the hidden primary and our secondaries serve the zone; the zone holding the NS names only needs a DNS API somewhere (registrar or any DNS provider, independent of the OpenStack provider). Internal names could move from
/etc/hoststo knot as well.