Skip to content

Secrets Management Architecture

Overview

ethere uses sops + age to encrypt all secrets (PKI certificates, cloud credentials, sensitive configs). Encrypted secrets are versioned in Git; private age keys stay outside the repository.

We use multi-recipient age encryption: each secret is encrypted for multiple recipients (human admins + Kubernetes operator). Each has their own private age key and can decrypt independently. This eliminates key sharing and simplifies revocation (remove a public key from .sops.yaml and re-encrypt).

Separate Git Repositories

A cluster uses two distinct Git repos:

  • cluster-root (main repo): infrastructure code (Ansible, Helm charts, operator source)
  • cluster-secrets (secrets repo): encrypted secrets and .sops.yaml only

Why separate? The Kubernetes operator must write to Git to automate secret rotation (certificates, credentials). With a single repo, the deploy key would have write access to all code, creating a major risk if the operator is compromised. Separation enforces least privilege: the operator can modify secrets but never infrastructure code.

Automated Rotation Workflow

The operator detects expiring certificates/credentials (via CronJob or events), generates new secrets, encrypts with sops, then performs git clone → modify .enc.yaml → git commit → git push to ethe-secrets. On conflict (concurrent operations), the push fails cleanly and alerts are sent. The GitOps controller (ArgoCD/Flux) detects changes and applies the new configuration automatically.

Operator Key Generation

The operator's private age key is generated in-memory by Ansible during cluster bootstrap. It's never written to disk: Ansible creates it, injects it directly into a Kubernetes Secret, adds the corresponding public key to .sops.yaml, then clears it from memory. Only the K8s Secret (protected by encrypted etcd + strict RBAC + NetworkPolicy) retains this private key. Human admins keep their own private age keys on their workstations (never versioned), ensuring they can always decrypt all secrets even if the cluster is destroyed.

Provider credentials: one root secret per admin (design, not implemented yet)

Each admin holds a single root secret: their age key, plus, per provider, a personal administrative credential (OVHcloud API token, OpenStack user…) sops-encrypted for that admin only. Every other credential is derived and scoped to the strict minimum:

  • OpenStack application credentials (restricted roles, access rules, expiration) for each component: the ethere Cluster API provider, the Cinder CSI driver, etc.;
  • S3 users with a policy limited to one bucket (etcd backups, CloudNativePG/Barman, the Zot registry, the git repositories' bundles: git_bundle_bucket, written today with the admin's own AWS profile, git.md);
  • DNS: the operator's API credentials scoped to the provider zones it writes (see dns.md, "Who manages what": the delegation and the NS names' addresses are written by an Ansible adapter per DNS provider, provisioning/dns.yml, not by OpenTofu).

Ansible adapters manage this layer, no OpenTofu (decided 2026-10-09, like the DNS delegation: dns.md, "Who manages what"): one Ansible module per capability and per provider, with the same arguments for every provider of a capability, e.g. ovh_s3_user / <provider>_s3_user or openstack.cloud.application_credential. The first one exists (2026-10-29, the etcd backups' user, playbooks/etcd/backup.yml): ovh_s3_user takes the provider's credentials, the project, the user's description and its S3 policy, and returns access and secret; the backups keep them in the cluster's Secret only, not yet in a secrets repository. The second (2026-10-31, playbooks/k8s/volumes.yml): the Cinder CSI driver's OpenStack application credential (openstack.cloud.application_credential), kept in the cluster's Secret only too (docs/admin/volumes.md). Supporting a new provider means adding its modules, never touching the callers: the "no OVH-only assumption" rule holds in this layer too. A playbook runs them with the admin's root credentials (sops-decrypted, passed as module arguments with no_log, never written to a file), and sops-encrypts the derived credentials into the secrets repository for the admins and the operator.

  • No state: what exists is found by name (e.g. an S3 user whose description is ethere:<cluster>:<bucket>, an application credential named after its component), so a run is idempotent and converges like ovh_dns_records does for records; check mode and diff show the plan.
  • A secret is only returned at creation: the module hands it over at once, and the playbook sops-encrypts it in the same run. If that step fails, or the encrypted copy is lost, the credential is replaced (a new one created, the old one revoked), never read back.
  • Plaintext never reaches the disk (the lesson of an earlier Terraform prototype, which left a decrypted state and a plaintext backend file behind after a failed run): the root credentials and the derived secrets only go through module arguments and sops' standard input.
  • Rotation stays with the operator: the playbook lays the identities and permissions; the operator rotates the derived credentials with its own scoped credential and never needs an admin's root secret.
  • Provider coverage: the OVHcloud API covers S3 users (project users, their S3 credentials, a policy per user) and DNS zones, openstack.cloud covers application credentials; for the other candidates of providers.md, an API covering identities and scoped credentials becomes a selection criterion.
  • Considered and dropped: OpenTofu (state holding the generated secrets, its client-side encryption, a bootstrap state outside the bucket it creates, a second tool next to Ansible), for the same reasons as for the DNS delegation.