Skip to content

Tenants

Applications of other projects (a tenant being a project, a repository deploying into the cluster, unlike the cluster's configuration, e.g. dampetra, which describes the cluster) run in namespaces the cluster grants them. The cluster grants, the project deploys: the cluster knows what a tenant may use (its namespaces, the names its routes may take, its quotas, its members), never what runs there (no chart, no version, no notion of environment: a tenant's staging is one more namespace).

the cluster's configuration (e.g. dampetra)      the tenant's project
  tenants/ (a chart: ethere-tenants + values)      its chart, its values per target, its Flux objects
        │ git                                             │ git (or OCI)
        ▼                                                 ▼
  Flux (flux-system): HelmRelease tenants ──▶ namespaces, quotas, bindings, allowed names
                                              Flux in the tenant's namespace: its HelmRelease ──▶ its application

Common tasks

Every change is a commit to tenants/values.yaml in the cluster's configuration, pushed: Flux applies it within k8s_flux_tenants_interval (a minute). Nothing to run.

A new tenant, or a namespace for one: its entry under tenants, each namespace with the names its routes may take and its quota:

tenants:
  liveguard:
    namespaces:
      liveguard-staging: {hostnames: [staging.liveguard.io]}
      liveguard: {hostnames: [app.liveguard.io, "*.liveguard.io"], quota: {requests.cpu: "8", requests.memory: 16Gi}}
    members:
      - {user: alice@example.com, role: admin, namespaces: [liveguard-staging, liveguard]}
      - {serviceAccount: ci, role: deployer, namespaces: [liveguard-staging]}

Names outside the cluster's zones (app.liveguard.io) also need their listener and their provider's records (Gateway). The tenant then deploys with its own Flux objects (the Users tab: Deploying).

A member: an entry of members, a person by their email (user), a CI by a service account made in the namespace (serviceAccount), role: admin or deployer, for some of the tenant's namespaces. A person also needs an identity: their user in the cluster's identities, and a login code for their first security key (Identities).

Removing a namespace or a tenant: its entry removed. Its quota and bindings go, the namespace stays with its data (helm.sh/resource-policy: keep): delete it yourself once sure (kubectl delete namespace <name>).

Removing a member: their entry removed (their bindings go); their identity too (Identities).

Did it apply?

kubectl -n flux-system get gitrepository,helmrelease tenants   # the revision Flux has, the release's state
kubectl -n flux-system logs deploy/helm-controller             # a refusal the chart or the lockdown made
kubectl get namespace -l ethe.re/tenant=<tenant> -o yaml       # a tenant's namespaces, their allowed names

The chart refuses, naming the mistake, a namespace in two tenants, a member's namespace of another tenant, an unknown role, the service account flux as a member: the release stays as it was. Quote names YAML would read otherwise ("y", "no", "on", "*.example.com"): Helm reads YAML 1.1.

After moving the cluster to a newer ethere: etherectl tenants update tenants copies its library chart into the declaration, committed with the new ref.

Setting it up

Once per cluster:

  1. The declaration: etherectl tenants init tenants makes it in the cluster's configuration: Chart.yaml, a one-line template including the library chart ethere-tenants (charts/ethere-tenants in ethere, copied into tenants/charts/: the declaration depends on its own directory alone), and values.yaml, the only file to edit.
  2. Flux reading it: the repository in the cluster's inventory (k8s_flux_tenants_url, _branch, _path (tenants), its credentials k8s_flux_tenants_credentials: a sops-encrypted JSON file of the cluster's configuration, the Secret's data as Flux reads it, username/password over HTTPS or identity/known_hosts over SSH), then an admin's run of the configuration, which writes the repository's Secret (cluster_credentials: it is encrypted for the admins, the operator never writes it); k8s/flux.yml alone applies just Flux's part:

    ansible-playbook $ETHERE_ROOT_DIR/playbooks/k8s/configure.yml
    

What the chart makes

Per namespace:

  • the Namespace, labelled ethe.re/tenant: <tenant>, its allowed names in the annotation ethe.re/hostnames, and helm.sh/resource-policy: keep (above);
  • its ResourceQuota ethere-tenant, quota being the quota's hard (per namespace: a quota across a tenant's namespaces isn't possible with ResourceQuota);
  • a service account flux bound to admin: the one the tenant's own Flux releases name;
  • the ConfigMap ethere-docs: the namespace's own page for its members (its names, quota, members, the Flux objects to start from; docsUrl names the documentation's site).

Per member and namespace, a RoleBinding: role: admin → the ClusterRole admin (everything in the namespace, binding roles there included, but its quota; never the namespace itself), role: deployer → edit; to the user as the apiserver names it (oidc: + the user, userPrefix in the values) or to a service account made in the namespace (a CI's). admin and edit include the tenants' routes (HTTPRoute, GRPCRoute) and Cilium's network policies, which Kubernetes' own roles don't cover (ethere-tenant-edit, ethere-tenant-view: aggregated ClusterRoles of inventory/group_vars/k8s/tenants.yml).

Flux, locked down

Flux is an addon (inventory/group_vars/k8s/flux.yml: the fluxcd-community/flux2 chart, its images embedded, on the control plane): its source and helm controllers, nothing else, in flux-system (out of the default network policies: it reaches the git repositories). Locked down for tenants:

  • no object references another namespace's (--no-cross-namespace-refs);
  • a HelmRelease naming no service account impersonates its namespace's default, which has no rights.

A release breaking either rule is never installed; a cross-namespace reference is reported in helm-controller's log only (no status, no event).

k8s/flux.yml (part of k8s/configure.yml, which the operator runs too) applies the entry point: the GitRepository tenants and a HelmRelease tenants building the chart at k8s_flux_tenants_path, with reconcileStrategy: Revision (a commit changing only values.yaml is applied) and the service account ethere-tenants (cluster-admin: it makes namespaces and binds admin).

The names a tenant's routes take

The Gateway's HTTPS listeners accept routes from every namespace, and ExternalDNS publishes each accepted route's names. In a tenant's namespace (labelled ethe.re/tenant), a native ValidatingAdmissionPolicy (ethere-tenant-hostnames, inventory/group_vars/k8s/tenants.yml: no webhook to keep running) refuses an HTTPRoute or GRPCRoute:

  • without spec.hostnames (it would take every name of its listener);
  • with a name that isn't one of the namespace's ethe.re/hostnames, or under one of its *. entries (*.example.com allows a.example.com, *.a.example.com, not example.com).

The annotation is written by the chart; the members (admin in the namespace) can't change it: the namespace itself isn't theirs. Names outside the cluster's zones also need their listener and their provider's records (k8s_gateway_hostnames, dns_provider_zones: Gateway).

The tenant's side

Written for the tenants, served by the cluster: the Users tab (getting started, deploying, exposing, network…), and each namespace's own page (its ConfigMap ethere-docs, above). examples/tenant-app, a minimal application, shows each of the cluster's choices. Their images: the public registry, a repository pattern per group of the identity provider (Registry).

Not yet

CloudNativePG (PostgreSQL) then Valkey as data services for the tenants; names outside the cluster's zones declared by the chart; the registry's rights derived from this declaration rather than a list of their own (roadmap).