Volumes¶
Applications get persistent storage as PersistentVolumeClaims of the default StorageClass cinder
(k8s_volumes_storage_class): Cinder volumes, created by the Cinder CSI driver, attached by Nova to the server whose
node runs the pod. A volume outlives the pod and the server: when a server is replaced (provisioning/replace.yml
drains its node), its pods move to another node and their volumes are detached, then attached to that node's server.
Every image rollout replaces every server, so local disks are no place for data: they go with the server.
What it means for an application: docs/storage.md, served by the cluster (docs/admin/documentation.md).
The driver¶
The Cinder CSI driver of cloud-provider-openstack (inventory/group_vars/k8s/cinder-csi.yml, chart
openstack-cinder-csi matching the Kubernetes minor version) is an addon like Cilium: its chart is rendered into the
control plane's image and applied at boot, its container images are embedded (the controller's sidecars on the
control plane, the plugin on every node). In kube-system:
- the controller (a Deployment on the control plane) creates, deletes, attaches and resizes volumes through the
OpenStack API. It runs on the host's network: the pods are IPv6-only without 464XLAT (
docs/admin/egress.md), and an OpenStack catalog may give IPv4 addresses (the test cloud's does); like every host network pod it uses the node's resolvers. The chart has no setting for this: a kustomize patch of the rendered chart (patchesofk8s_addons_charts,images/_addons.yml) sets it, and moves its liveness port off the node plugin's (both on the host's network of every aio server); - the node plugin (a DaemonSet on every node, the host's network) finds the attached disk
(
/dev/disk/by-id/virtio-<id>: udev), formats it (ext4) and mounts it for the kubelet. It registers its node (CSINode) with the Nova server's ID, which the controller attaches the volumes to, and its availability zone: it learns both from the metadata service alone (no config drive), which the nodes leave open to their own processes and host network pods (k8s_node_metadata_block: only what they forward to it is dropped; the pods are denied it by Cilium too,docs/admin/network-policies.md).
The StorageClass, written by playbooks/k8s/volumes.yml (not an addon: an image serves every cloud, the volume type
is a cloud's): WaitForFirstConsumer (the volume is created once the pod is scheduled, in the zone of its node: the
driver reports each node's availability zone), Delete (a claim's volume goes with it), expandable. Its volume type
is k8s_volumes_type, a cloud profile's setting (empty: the cloud's default type); a StorageClass's parameters can't
change, so another type replaces it (the existing volumes keep theirs). The type must exist in every zone of the
nodes: OVHcloud's default, classic-multiattach, only exists in the zone any, and a volume of a node's zone fails.
No snapshots yet: neither their CRDs nor their controller.
Its credentials¶
The driver reads a cloud.conf from the Secret k8s_volumes_secret (cinder-csi-cloud-config), which
playbooks/k8s/volumes.yml writes from the workstation (provisioning/cluster.yml runs it): an OpenStack
application credential of the admin running it, named ethere:<cluster>:<region>:cinder-csi, scoped to the
cluster's project with the admin's roles, not allowed to create other credentials. Its secret exists only in the
cluster's Secret: Keystone shows it once, so the playbook does nothing while the Secret exists, and
-e k8s_volumes_rotate=true replaces both (the driver's pods restarted). Until the Secret exists, the driver's pods
wait for it. This is the second adapter of docs/admin/secrets.md (scoped credentials derived from the admin's); a
credential belongs to the user who created it: when that admin's OpenStack user goes, run the playbook again as
another (k8s_volumes_rotate).
Testing¶
tests/cluster/volumes.test.js (the test cloud): a claim written on one server and read on another, then a node
replaced by provisioning/replace.yml under a Deployment using it. The test cloud's Cinder: LVM in the qemu backend,
NFS in the lxd one (the container's own NFS server: an iSCSI attachment needs device nodes and udev, which a
container doesn't get).