Skip to content

Replicated git repositories

The cluster's git repositories, replicated on the git_replica servers (the images carrying etcd: ethere-k8s-aio, ethere-quorum), written from outside through the edge: today one repository, the PKI's (pki.md); later others (the secrets kept outside Kubernetes, TODO.md). Served by sys-apps/ethere-git-server. A crashed cluster is rebuilt, not repaired: after a crash, the only need is to get the repositories back (S3 bundles, below).

Replicas on the nodes, served over HTTPS with mutual TLS

  • Every server of the role git_replica holds a full copy (a bare repository in /var). The role goes on the images carrying etcd (ethere-k8s-aio, ethere-quorum): an odd number of stable servers, the same failure domains as etcd's quorum.
  • Git's smart HTTP protocol served by ethere-git-server, a small Go program packaged in ethere's repository (like sys-apps/ethere-log): TLS with a client certificate required (crypto/tls), git http-backend run as a CGI (net/http/cgi), the client certificate's subject passed as REMOTE_USER; only the standard library. No sshd on the images, as everywhere, and no gitolite: with a handful of identities, the authorization fits in the server (a push refused before git runs) and a pre-receive hook (below).
  • Identities, all from a dedicated CA pki (monthly rotation like the others):
    • each replica: a serving certificate (pki-server: the anycast address, its Git-Net address, its names in the internal zone, <server>.git.ethere.internal and writer.git.ethere.internal: git_names_domain, git_writer_name) and a client certificate (pki-replica, CN=<server>), both enrolled through the console like every other node certificate (keys in /var/lib/ethere/enroll, never leaving the server);
    • the signer: a client certificate (O=ethere:git-signer, CN=<admin>), one-time: every playbook acting as the signer (git/seed.yml, git/push.yml, git/writer.yml) generates a fresh key and has it signed on the workstation by the same CA (utils/_signer_certificate.yml: its key decrypted by sops, through openssl's standard input; backdated by 10 minutes, valid a couple of hours), kept in tmpfs only (a private directory of $XDG_RUNTIME_DIR/ethere/<cluster>/, removed when the playbook ends); the operator gets its own.
    • Both ends check the other against the CA pki's bundle (previous, current and next months): the nodes get it with the user-data, like the other bundles.
  • Reads (any certificate of the CA) inside the cluster: through an anycast address on a dedicated private network (Git-Net), as for every internal service. Writes go to the writer (below).
  • No public port on the replicas: everything coming from the Internet goes through the edge (HAProxy, role edge: on ethere-k8s-aio, or the ethere-edge image), the signer's traffic included. HAProxy relays in TCP mode, routing on the SNI, so the mutual TLS stays end to end (the edge holds no PKI key and sees nothing): writer.git.ethere.internal goes to the writer designated by DNS, <server>.git.ethere.internal to that replica (writer change), both resolved on every connection against the hidden primary (one rule, no per-replica configuration); the Kubernetes API's anycast name goes to the API (k8s/kubeconfig.yml, which the writer's playbook uses). The edge has ports on Git-Net, DNS-Net and K8s-Nodes-Net.
  • Writes, the writer check and the replication never depend on etcd or Kubernetes: ethere-git-server, git, the hooks and HAProxy are plain systemd units, so a PKI outage and an etcd outage can't block each other (etcd gets its certificates from this PKI).

One writer, designated by DNS

  • Exactly one replica accepts pushes to refs/heads/main: the writer, designated by a record of the internal zone ethere.internal (dns_internal_zone: unsigned, served by the hidden primary alone on DNS-Net, never transferred to the secondaries; dns.md), writer.git.ethere.internal AAAA: the writer's Git-Net address (HAProxy resolves A/AAAA/SRV, not TXT). Each replica's name, <server>.git.ethere.internal, is published the same way. The names are fixed (git_names_domain, git_writer_name, all/networks.yml): nothing comes with the user-data.
  • The records are Kubernetes objects: DNSEndpoint objects in the namespace dns (git-writer, git-replica-<server>), which ExternalDNS publishes into the primary (dns.md): they survive the primary's moves. They are written by the playbook git/writer.yml (below) through the admin kubeconfig (k8s/kubeconfig.yml, the API through the edge): by an admin today, by the operator later, the same procedure. Nothing outside the cluster updates the primary: ExternalDNS does, with its own certificate.
  • The edge's HAProxy resolves the git names against the hidden primary directly (its resolvers section, DNS-Net's anycast address, 1 s hold times, do-resolve on the SNI): it routes the writer's name to whichever replica the record designates, and drops the connection while the record is absent.
  • Each replica accepts a push to main only if the hidden primary (queried directly over TCP at DNS-Net's anycast address, without recursion: never a cached answer) designates its own Git-Net address, and only it: HAProxy's routing is not trusted for that, it could act on an older answer. One source, no propagation delay. While the primary (a pod) is down there is no writer and no write: it moves to another control plane server 30 s after its server stops answering, and ExternalDNS (moved likewise) publishes the records again. With ethere's architecture a crashed cluster is rebuilt, not repaired, and the repositories are recovered from their S3 bundles (below). A client pushing to a replica that isn't the writer is refused and resolves again. The replication follows the same answer: a replica fetches from the address the record designates, and waits while there is none.
  • Writes are fast-forward only, from one signer at a time: no conflicts, and git's atomic ref update makes each write build on the latest state.

Replication: pull, with acknowledgements

  • Each replica fetches main from the writer (by name) every few seconds; the writer never needs to know the replicas.
  • After each fetch, a replica records what it holds by pushing refs/replicas/<its name> to the writer: the hook lets a pki-replica identity update its own ref only. The signer waits until a majority of the replicas (the cluster's servers of the group git_replica in OpenStack) acknowledged a write, then uploads its bundle (below); it keeps the write locally until then: a writer dying right after accepting a push loses nothing the signer doesn't still have.
  • A new or rebuilt replica fetches from the writer and serves reads only once it holds main; it never accepts writes before being designated.

Changing the writer

Phased, like an instance rotation, run by ansible-playbook $ETHERE_ROOT_DIR/playbooks/git/writer.yml [-e git_writer_server=<name>] (an admin today, the operator later; the edge from edge_address/edge_address_port, else the oldest edge). What the primary serves is asked from inside its pod (kdig in the knot container, kubectl exec): the internal zone is on no secondary.

  1. Publish every replica's name (the objects of replicas gone deleted), and wait until the primary serves them.
  2. If the writer designated is a reachable replica holding the most advanced main (and git_writer_server names no other), keep it: nothing else to do.
  3. Otherwise remove the record and wait until the primary serves none: no replica accepts writes, the branches stop moving.
  4. Fetch main from every replica (through the edge, by their names, with a one-time signer certificate from the CA pki; the module git_replica_heads) and designate one holding the most advanced: every other replica's main is an ancestor of it, so they fast-forward to it (a replica follows the writer with a forced fetch: designating a replica that is behind would roll the others back). git_writer_server is refused if that replica is behind or unreachable. Nothing is pushed: no replica ever accepts a write without being designated.
  5. Point the record to it, and wait until the primary serves it.

Same procedure when the writer dies: its record designates an address no replica answers, and the most advanced survivor is designated. A write the dead writer accepted but no survivor holds is not lost: it was durable only once a majority acknowledged it, and the signer still has it (git/push.yml again).

Writing the repository, and its S3 bundles

ansible-playbook $ETHERE_ROOT_DIR/playbooks/git/seed.yml (the first write: makes pki_dir a git repository, branch main, if it isn't one) and git/push.yml (later writes, -e git_message=…: fast-forwards pki_dir to the writer's main first), sharing git/_write.yml, run on the workstation by an admin (the operator later, the same playbooks):

  1. Commit what changed in pki_dir and push main to writer.git.ethere.internal through the edge (edge_address/edge_address_port, else the oldest edge: utils/_edge.yml), with a one-time signer certificate (above). The writer must be designated first (git/writer.yml).
  2. Wait until a majority of the replicas acknowledged the head (refs/replicas/*; the replicas counted in OpenStack, git_replica_count overrides; git_write_timeout, 300 s).
  3. Upload a git bundle of main (git_bundle_key, pki.bundle) to the versioned S3 bucket git_bundle_bucket (<cluster>-git; inventory/group_vars/all/git.yml), created and versioned if missing (amazon.aws). A write is durable only once its bundle is uploaded. No credentials on the servers: the writers upload, not the replicas.

Without an S3 outside the cluster (git_bundle_in_cluster: a standalone cluster's, docs/admin/standalone.md), the bundle is the Secret kube-system/git-bundle instead, one version, gone with the cluster: an operator's first clone (operator/pki.yml, its ServiceAccount has the right to read that Secret alone), not a recovery path.

ansible-playbook $ETHERE_ROOT_DIR/playbooks/git/restore.yml [-e git_bundle_version=<id>] gets the repository back from the newest bundle (or an older version of the bucket): cloned into a missing or empty pki_dir, fast-forwarded into an existing repository (refused if it diverged, or if pki_dir isn't a repository). It needs only the bucket and the admin's credentials: no cluster, no edge, no DNS (a rebuild after a crash).

The bucket is reached with the admin's AWS profile (AWS_PROFILE, the cluster's aws/config, whose S3 endpoint must be declared in a services section: boto3 reads that, not the nested s3 = endpoint_url form) until a derived S3 user limited to the bucket exists (secrets.md). Recovery reads the bucket with the admin's own credentials, never with credentials stored in the repositories it recovers.

Increments

  1. Replicas with a fixed writer (done: inventory/group_vars/git_replica/, sys-apps/ethere-git-server, tests/cluster/pki.test.js; the writer is the replica that bootstrapped, written in /run/ethere/hosts by the user-data, or pki_writer_ip): ethere-git-server (package, unit), the role git_replica on the etcd-carrying images, Git-Net and its anycast, the CA pki and its enrolled certificates, the writer given by the provisioning instead of DNS, pull replication with acknowledgements. Cluster test on the test cloud, the signer being a probe on Git-Net: seed, a write through the writer, a push refused on another replica, a majority of acknowledgements, a rebuilt replica catching up.
  2. The edge in front (done: inventory/group_vars/edge/, image ethere-edge, lib/pki.js, replaced by git/seed.yml/git/push.yml in the redesign's P6): the role edge (HAProxy; the exposure's design in TODO.md) with the PKI's SNI routes (the writer and the anycast address; the per-replica routes come with increment 3, which needs them), etherectl pki seed/push from the workstation through it.
  3. Writer designated by DNS (done: etherectl pki writer, lib/dns.js, ethere-git-server 0.4.1, the edge's do-resolve, the primary's TLS listener and pki-signer ACLs, the names of replicas gone removed): the AAAA records, HAProxy's resolution and the replicas' check against the hidden primary, the writer change procedure. Cluster test: a secondary, the primary on a control plane server, three replicas and an edge; designation, seed, a writer change, the writer's death, a new replica catching up. Needs the zone's authoritative servers (ethere-dns-secondary) in the topology. Superseded by the redesign's P5: the names in the internal zone, DNSEndpoint objects published by ExternalDNS, git/writer.yml instead of etherectl pki writer, no pki-signer ACLs nor dns-primary route on the edge; cluster test with two control plane servers, the primary moved by stopping its server.
  4. servers.yml on the cluster's repository (done, the redesign's P7): once pki_dir is marked as the cluster's clone (git config ethere.cluster true, set by git/seed.yml and git/restore.yml), provisioning/servers.yml fast-forwards it to the writer's main before signing, and pushes what it changed after, waiting for a majority of acknowledgements, then uploading the bundle (git/_write.yml); pki_git_repo and git_acp removed.

Open questions

  • How the operator authenticates (its signer certificate, its age key) once it exists.
  • Several repositories: paths, access per identity, one writer for all or one per repository (TODO.md).