Tenants¶
Applications of other projects (a tenant being a project, a repository deploying into the cluster, unlike the
cluster's configuration, e.g. dampetra, which describes the cluster) run in namespaces the cluster grants them. The
cluster grants, the project deploys: the cluster knows what a tenant may use (its namespaces, the names its routes may
take, its quotas, its members), never what runs there (no chart, no version, no notion of environment: a tenant's
staging is one more namespace).
the cluster's configuration (e.g. dampetra) the tenant's project
tenants/ (a chart: ethere-tenants + values) its chart, its values per target, its Flux objects
│ git │ git (or OCI)
▼ ▼
Flux (flux-system): HelmRelease tenants ──▶ namespaces, quotas, bindings, allowed names
Flux in the tenant's namespace: its HelmRelease ──▶ its application
Common tasks¶
Every change is a commit to tenants/values.yaml in the cluster's configuration, pushed: Flux applies it within
k8s_flux_tenants_interval (a minute). Nothing to run.
A new tenant, or a namespace for one: its entry under tenants, each namespace with the names its routes may
take and its quota:
tenants:
liveguard:
namespaces:
liveguard-staging: {hostnames: [staging.liveguard.io]}
liveguard: {hostnames: [app.liveguard.io, "*.liveguard.io"], quota: {requests.cpu: "8", requests.memory: 16Gi}}
members:
- {user: alice@example.com, role: admin, namespaces: [liveguard-staging, liveguard]}
- {serviceAccount: ci, role: deployer, namespaces: [liveguard-staging]}
Names outside the cluster's zones (app.liveguard.io) also need their listener and their provider's records
(Gateway). The tenant then deploys with its own Flux objects (the Users tab: Deploying).
A member: an entry of members, a person by their email (user), a CI by a service account made in the
namespace (serviceAccount), role: admin or deployer, for some of the tenant's namespaces. A person also needs
an identity: their user in the cluster's identities, and a login code for their first security key
(Identities).
Removing a namespace or a tenant: its entry removed. Its quota and bindings go, the namespace stays with its
data (helm.sh/resource-policy: keep): delete it yourself once sure (kubectl delete namespace <name>).
Removing a member: their entry removed (their bindings go); their identity too (Identities).
Did it apply?
kubectl -n flux-system get gitrepository,helmrelease tenants # the revision Flux has, the release's state
kubectl -n flux-system logs deploy/helm-controller # a refusal the chart or the lockdown made
kubectl get namespace -l ethe.re/tenant=<tenant> -o yaml # a tenant's namespaces, their allowed names
The chart refuses, naming the mistake, a namespace in two tenants, a member's namespace of another tenant, an
unknown role, the service account flux as a member: the release stays as it was. Quote names YAML would read
otherwise ("y", "no", "on", "*.example.com"): Helm reads YAML 1.1.
After moving the cluster to a newer ethere: etherectl tenants update tenants copies its library chart into the
declaration, committed with the new ref.
Setting it up¶
Once per cluster:
- The declaration:
etherectl tenants init tenantsmakes it in the cluster's configuration:Chart.yaml, a one-line template including the library chartethere-tenants(charts/ethere-tenantsin ethere, copied intotenants/charts/: the declaration depends on its own directory alone), andvalues.yaml, the only file to edit. -
Flux reading it: the repository in the cluster's inventory (
k8s_flux_tenants_url,_branch,_path(tenants), its credentialsk8s_flux_tenants_credentials: a sops-encrypted JSON file of the cluster's configuration, the Secret's data as Flux reads it,username/passwordover HTTPS oridentity/known_hostsover SSH), then an admin's run of the configuration, which writes the repository's Secret (cluster_credentials: it is encrypted for the admins, the operator never writes it);k8s/flux.ymlalone applies just Flux's part:
What the chart makes¶
Per namespace:
- the
Namespace, labelledethe.re/tenant: <tenant>, its allowed names in the annotationethe.re/hostnames, andhelm.sh/resource-policy: keep(above); - its
ResourceQuotaethere-tenant,quotabeing the quota'shard(per namespace: a quota across a tenant's namespaces isn't possible withResourceQuota); - a service account
fluxbound toadmin: the one the tenant's own Flux releases name; - the ConfigMap
ethere-docs: the namespace's own page for its members (its names, quota, members, the Flux objects to start from;docsUrlnames the documentation's site).
Per member and namespace, a RoleBinding: role: admin → the ClusterRole admin (everything in the namespace,
binding roles there included, but its quota; never the namespace itself), role: deployer → edit; to the user as
the apiserver names it (oidc: + the user, userPrefix in the values) or to a service account made in the
namespace (a CI's). admin and edit include the tenants' routes (HTTPRoute, GRPCRoute) and Cilium's network
policies, which Kubernetes' own roles don't cover (ethere-tenant-edit, ethere-tenant-view: aggregated
ClusterRoles of inventory/group_vars/k8s/tenants.yml).
Flux, locked down¶
Flux is an addon (inventory/group_vars/k8s/flux.yml: the fluxcd-community/flux2 chart, its images embedded, on
the control plane): its source and helm controllers, nothing else, in flux-system (out of the default network
policies: it reaches the git repositories). Locked down for tenants:
- no object references another namespace's (
--no-cross-namespace-refs); - a
HelmReleasenaming no service account impersonates its namespace'sdefault, which has no rights.
A release breaking either rule is never installed; a cross-namespace reference is reported in helm-controller's log only (no status, no event).
k8s/flux.yml (part of k8s/configure.yml, which the operator runs too) applies the entry point: the
GitRepository tenants and a HelmRelease tenants building the chart at k8s_flux_tenants_path, with
reconcileStrategy: Revision (a commit changing only values.yaml is applied) and the service account
ethere-tenants (cluster-admin: it makes namespaces and binds admin).
The names a tenant's routes take¶
The Gateway's HTTPS listeners accept routes from every namespace, and ExternalDNS publishes each accepted route's
names. In a tenant's namespace (labelled ethe.re/tenant), a native ValidatingAdmissionPolicy
(ethere-tenant-hostnames, inventory/group_vars/k8s/tenants.yml: no webhook to keep running) refuses an
HTTPRoute or GRPCRoute:
- without
spec.hostnames(it would take every name of its listener); - with a name that isn't one of the namespace's
ethe.re/hostnames, or under one of its*.entries (*.example.comallowsa.example.com,*.a.example.com, notexample.com).
The annotation is written by the chart; the members (admin in the namespace) can't change it: the namespace
itself isn't theirs. Names outside the cluster's zones also need their listener and their provider's records
(k8s_gateway_hostnames, dns_provider_zones: Gateway).
The tenant's side¶
Written for the tenants, served by the cluster: the Users tab (getting started, deploying, exposing, network…), and
each namespace's own page (its ConfigMap ethere-docs, above). examples/tenant-app, a minimal application, shows
each of the cluster's choices. Their images: the public registry, a repository pattern per group of the identity
provider (Registry).
Not yet¶
CloudNativePG (PostgreSQL) then Valkey as data services for the tenants; names outside the cluster's zones declared by the chart; the registry's rights derived from this declaration rather than a list of their own (roadmap).