Replicated git repositories¶
The cluster's git repositories, replicated on the git_replica servers (the images carrying etcd: ethere-k8s-aio, ethere-quorum), written from outside through the edge: today one repository, the PKI's (pki.md); later others (the secrets kept outside Kubernetes, TODO.md). Served by sys-apps/ethere-git-server. A crashed cluster is rebuilt, not repaired: after a crash, the only need is to get the repositories back (S3 bundles, below).
Replicas on the nodes, served over HTTPS with mutual TLS¶
- Every server of the role
git_replicaholds a full copy (a bare repository in/var). The role goes on the images carrying etcd (ethere-k8s-aio,ethere-quorum): an odd number of stable servers, the same failure domains as etcd's quorum. - Git's smart HTTP protocol served by
ethere-git-server, a small Go program packaged in ethere's repository (likesys-apps/ethere-log): TLS with a client certificate required (crypto/tls),git http-backendrun as a CGI (net/http/cgi), the client certificate's subject passed asREMOTE_USER; only the standard library. No sshd on the images, as everywhere, and no gitolite: with a handful of identities, the authorization fits in the server (a push refused before git runs) and apre-receivehook (below). - Identities, all from a dedicated CA
pki(monthly rotation like the others):- each replica: a serving certificate (
pki-server: the anycast address, its Git-Net address, its names in the internal zone,<server>.git.ethere.internalandwriter.git.ethere.internal:git_names_domain,git_writer_name) and a client certificate (pki-replica,CN=<server>), both enrolled through the console like every other node certificate (keys in/var/lib/ethere/enroll, never leaving the server); - the signer: a client certificate (
O=ethere:git-signer, CN=<admin>), one-time: every playbook acting as the signer (git/seed.yml,git/push.yml,git/writer.yml) generates a fresh key and has it signed on the workstation by the same CA (utils/_signer_certificate.yml: its key decrypted by sops, through openssl's standard input; backdated by 10 minutes, valid a couple of hours), kept in tmpfs only (a private directory of$XDG_RUNTIME_DIR/ethere/<cluster>/, removed when the playbook ends); the operator gets its own. - Both ends check the other against the CA
pki's bundle (previous, current and next months): the nodes get it with the user-data, like the other bundles.
- each replica: a serving certificate (
- Reads (any certificate of the CA) inside the cluster: through an anycast address on a dedicated private network (
Git-Net), as for every internal service. Writes go to the writer (below). - No public port on the replicas: everything coming from the Internet goes through the edge (HAProxy, role
edge: onethere-k8s-aio, or theethere-edgeimage), the signer's traffic included. HAProxy relays in TCP mode, routing on the SNI, so the mutual TLS stays end to end (the edge holds no PKI key and sees nothing):writer.git.ethere.internalgoes to the writer designated by DNS,<server>.git.ethere.internalto that replica (writer change), both resolved on every connection against the hidden primary (one rule, no per-replica configuration); the Kubernetes API's anycast name goes to the API (k8s/kubeconfig.yml, which the writer's playbook uses). The edge has ports on Git-Net, DNS-Net and K8s-Nodes-Net. - Writes, the writer check and the replication never depend on etcd or Kubernetes:
ethere-git-server, git, the hooks and HAProxy are plain systemd units, so a PKI outage and an etcd outage can't block each other (etcd gets its certificates from this PKI).
One writer, designated by DNS¶
- Exactly one replica accepts pushes to
refs/heads/main: the writer, designated by a record of the internal zoneethere.internal(dns_internal_zone: unsigned, served by the hidden primary alone on DNS-Net, never transferred to the secondaries;dns.md),writer.git.ethere.internalAAAA: the writer's Git-Net address (HAProxy resolves A/AAAA/SRV, not TXT). Each replica's name,<server>.git.ethere.internal, is published the same way. The names are fixed (git_names_domain,git_writer_name,all/networks.yml): nothing comes with the user-data. - The records are Kubernetes objects:
DNSEndpointobjects in the namespacedns(git-writer,git-replica-<server>), which ExternalDNS publishes into the primary (dns.md): they survive the primary's moves. They are written by the playbookgit/writer.yml(below) through the admin kubeconfig (k8s/kubeconfig.yml, the API through the edge): by an admin today, by the operator later, the same procedure. Nothing outside the cluster updates the primary: ExternalDNS does, with its own certificate. - The edge's HAProxy resolves the git names against the hidden primary directly (its
resolverssection, DNS-Net's anycast address, 1 sholdtimes,do-resolveon the SNI): it routes the writer's name to whichever replica the record designates, and drops the connection while the record is absent. - Each replica accepts a push to
mainonly if the hidden primary (queried directly over TCP at DNS-Net's anycast address, without recursion: never a cached answer) designates its own Git-Net address, and only it: HAProxy's routing is not trusted for that, it could act on an older answer. One source, no propagation delay. While the primary (a pod) is down there is no writer and no write: it moves to another control plane server 30 s after its server stops answering, and ExternalDNS (moved likewise) publishes the records again. With ethere's architecture a crashed cluster is rebuilt, not repaired, and the repositories are recovered from their S3 bundles (below). A client pushing to a replica that isn't the writer is refused and resolves again. The replication follows the same answer: a replica fetches from the address the record designates, and waits while there is none. - Writes are fast-forward only, from one signer at a time: no conflicts, and git's atomic ref update makes each write build on the latest state.
Replication: pull, with acknowledgements¶
- Each replica fetches
mainfrom the writer (by name) every few seconds; the writer never needs to know the replicas. - After each fetch, a replica records what it holds by pushing
refs/replicas/<its name>to the writer: the hook lets apki-replicaidentity update its own ref only. The signer waits until a majority of the replicas (the cluster's servers of the groupgit_replicain OpenStack) acknowledged a write, then uploads its bundle (below); it keeps the write locally until then: a writer dying right after accepting a push loses nothing the signer doesn't still have. - A new or rebuilt replica fetches from the writer and serves reads only once it holds
main; it never accepts writes before being designated.
Changing the writer¶
Phased, like an instance rotation, run by ansible-playbook $ETHERE_ROOT_DIR/playbooks/git/writer.yml [-e git_writer_server=<name>] (an admin today, the operator later; the edge from edge_address/edge_address_port, else the oldest edge). What the primary serves is asked from inside its pod (kdig in the knot container, kubectl exec): the internal zone is on no secondary.
- Publish every replica's name (the objects of replicas gone deleted), and wait until the primary serves them.
- If the writer designated is a reachable replica holding the most advanced
main(andgit_writer_servernames no other), keep it: nothing else to do. - Otherwise remove the record and wait until the primary serves none: no replica accepts writes, the branches stop moving.
- Fetch
mainfrom every replica (through the edge, by their names, with a one-time signer certificate from the CApki; the modulegit_replica_heads) and designate one holding the most advanced: every other replica'smainis an ancestor of it, so they fast-forward to it (a replica follows the writer with a forced fetch: designating a replica that is behind would roll the others back).git_writer_serveris refused if that replica is behind or unreachable. Nothing is pushed: no replica ever accepts a write without being designated. - Point the record to it, and wait until the primary serves it.
Same procedure when the writer dies: its record designates an address no replica answers, and the most advanced survivor is designated. A write the dead writer accepted but no survivor holds is not lost: it was durable only once a majority acknowledged it, and the signer still has it (git/push.yml again).
Writing the repository, and its S3 bundles¶
ansible-playbook $ETHERE_ROOT_DIR/playbooks/git/seed.yml (the first write: makes pki_dir a git repository, branch main, if it isn't one) and git/push.yml (later writes, -e git_message=…: fast-forwards pki_dir to the writer's main first), sharing git/_write.yml, run on the workstation by an admin (the operator later, the same playbooks):
- Commit what changed in
pki_dirand pushmaintowriter.git.ethere.internalthrough the edge (edge_address/edge_address_port, else the oldest edge:utils/_edge.yml), with a one-time signer certificate (above). The writer must be designated first (git/writer.yml). - Wait until a majority of the replicas acknowledged the head (
refs/replicas/*; the replicas counted in OpenStack,git_replica_countoverrides;git_write_timeout, 300 s). - Upload a git bundle of
main(git_bundle_key,pki.bundle) to the versioned S3 bucketgit_bundle_bucket(<cluster>-git;inventory/group_vars/all/git.yml), created and versioned if missing (amazon.aws). A write is durable only once its bundle is uploaded. No credentials on the servers: the writers upload, not the replicas.
Without an S3 outside the cluster (git_bundle_in_cluster: a standalone cluster's, docs/admin/standalone.md), the bundle is the Secret kube-system/git-bundle instead, one version, gone with the cluster: an operator's first clone (operator/pki.yml, its ServiceAccount has the right to read that Secret alone), not a recovery path.
ansible-playbook $ETHERE_ROOT_DIR/playbooks/git/restore.yml [-e git_bundle_version=<id>] gets the repository back from the newest bundle (or an older version of the bucket): cloned into a missing or empty pki_dir, fast-forwarded into an existing repository (refused if it diverged, or if pki_dir isn't a repository). It needs only the bucket and the admin's credentials: no cluster, no edge, no DNS (a rebuild after a crash).
The bucket is reached with the admin's AWS profile (AWS_PROFILE, the cluster's aws/config, whose S3 endpoint must be declared in a services section: boto3 reads that, not the nested s3 = endpoint_url form) until a derived S3 user limited to the bucket exists (secrets.md). Recovery reads the bucket with the admin's own credentials, never with credentials stored in the repositories it recovers.
Increments¶
- Replicas with a fixed writer (done:
inventory/group_vars/git_replica/,sys-apps/ethere-git-server,tests/cluster/pki.test.js; the writer is the replica that bootstrapped, written in/run/ethere/hostsby the user-data, orpki_writer_ip):ethere-git-server(package, unit), the rolegit_replicaon the etcd-carrying images, Git-Net and its anycast, the CApkiand its enrolled certificates, the writer given by the provisioning instead of DNS, pull replication with acknowledgements. Cluster test on the test cloud, the signer being a probe on Git-Net: seed, a write through the writer, a push refused on another replica, a majority of acknowledgements, a rebuilt replica catching up. - The edge in front (done:
inventory/group_vars/edge/, imageethere-edge,lib/pki.js, replaced bygit/seed.yml/git/push.ymlin the redesign's P6): the roleedge(HAProxy; the exposure's design inTODO.md) with the PKI's SNI routes (the writer and the anycast address; the per-replica routes come with increment 3, which needs them),etherectl pki seed/pushfrom the workstation through it. - Writer designated by DNS (done:
etherectl pki writer,lib/dns.js,ethere-git-server0.4.1, the edge'sdo-resolve, the primary's TLS listener andpki-signerACLs, the names of replicas gone removed): the AAAA records, HAProxy's resolution and the replicas' check against the hidden primary, the writer change procedure. Cluster test: a secondary, the primary on a control plane server, three replicas and an edge; designation, seed, a writer change, the writer's death, a new replica catching up. Needs the zone's authoritative servers (ethere-dns-secondary) in the topology. Superseded by the redesign's P5: the names in the internal zone,DNSEndpointobjects published by ExternalDNS,git/writer.ymlinstead ofetherectl pki writer, nopki-signerACLs nordns-primaryroute on the edge; cluster test with two control plane servers, the primary moved by stopping its server. servers.ymlon the cluster's repository (done, the redesign's P7): oncepki_diris marked as the cluster's clone (git config ethere.cluster true, set bygit/seed.ymlandgit/restore.yml),provisioning/servers.ymlfast-forwards it to the writer'smainbefore signing, and pushes what it changed after, waiting for a majority of acknowledgements, then uploading the bundle (git/_write.yml);pki_git_repoandgit_acpremoved.
Open questions¶
- How the operator authenticates (its signer certificate, its age key) once it exists.
- Several repositories: paths, access per identity, one writer for all or one per repository (
TODO.md).