Skip to content

On-demand GPU nodes (design, not implemented yet)

GPU needs are occasional batch jobs: when a job needs a GPU, a GPU node must be created automatically, then deleted once no GPU work remains. GPU instances are expensive and billed per started hour (OVHcloud hourly billing), so nodes must exist only while needed, and their deletion must follow the billing. Start-up latency (a few minutes to boot, join and pull images) is not an issue for batch.

Ephemeral node pools managed by the operator

  • Jobs declare their requirements; the operator picks the cheapest available flavor that satisfies them (the "price expander" idea of autoscalers), instead of fixed pools with fallback orders.
    • Requirements are expressed with standard Kubernetes mechanisms, on the labels of NVIDIA GPU Feature Discovery (GFD): nvidia.com/gpu.memory (MiB), nvidia.com/gpu.product, nvidia.com/gpu.count, nvidia.com/gpu.compute.major/minor… Node affinity supports Gt/Lt on numeric label values, so a job can ask for "GPU memory > 40 GB" or "compute capability ≥ 8.9" (e.g. FP8 needs Ada/Hopper). Jobs are written as for any GFD-equipped cluster, and created nodes carry these labels.
    • Later: Dynamic Resource Allocation (DRA, stable since Kubernetes 1.34): ResourceClaims selecting devices with CEL expressions (memory, product, several GPUs on one node), with NVIDIA's DRA driver. Node affinity on GFD labels first, DRA kept in sight.
    • Requirements a job may express: memory per GPU, number of GPUs, architecture / compute capability, CPU and RAM, local NVMe scratch, region, maximum price.
  • Flavor catalog: per flavor, the GPU model, memory, architecture, compute capability, GPU count, CPU, RAM, local storage, hourly price and regions; built from the provider's API (prices, availability) and a table of GPU characteristics.
  • Scale-up: for each pod stuck in Pending because no node offers the requested GPU resource (Insufficient nvidia.com/gpu), the operator simulates the labels a node of each flavor would have (like Cluster Autoscaler's node templates), keeps the flavors satisfying the pod's constraints, and creates an instance of the cheapest available one, moving to the next one when out of stock or quota. Thanks to the zero post-boot configuration, the instance joins the cluster by itself, like an added ethere-k8s-node. GPU nodes carry a taint (e.g. nvidia.com/gpu=present:NoSchedule) so that only GPU workloads (which tolerate it) land on them.
  • A custom resource (e.g. GpuProvisioner, ethe.re/v1alpha1) holds the policy: allowed flavors, regions/AZ, maximum nodes and cost, image (ethere-k8s-node-gpu), labels and taints, scale-down policy.
  • Billing-aligned scale-down: since every started hour is billed, a node is never deleted after "one idle hour", but just before the end of each billed hour (e.g. at minute 55, counted from its creation):

    1. a node is idle when it runs no GPU workload;
    2. at the end of each billed hour, if idle: cordon it, check again that no workload was scheduled in between, then delete it;
    3. otherwise keep it for the next hour, which is due anyway.

    A started hour is never paid for nothing, and a job arriving within the hour reuses the node already paid for.

  • Safeguards: maximum nodes and cost per pool, creation timeout, fallback to another flavor or AZ when the provider is out of GPU capacity (it happens), alerts.

  • Packing several pending pods onto a multi-GPU flavor (e.g. l4-180 for two 1-GPU pods) can come later; one node per pending demand first.
  • The server creation logic is shared with the bootstrap (roles of the ethere collection, see the operator item in TODO.md).

Why not existing tools

  • Cluster Autoscaler does scale from zero, but on OpenStack it goes through Magnum or Cluster API, which impose their own bootstrap (kubeadm, cloud-init), incompatible with ethere's images. Its externalgrpc provider would let the operator only create/delete instances, but it can't align deletions with billed hours.
  • Karpenter has no OpenStack provider.

OVHcloud GPU flavors (catalog of 2026-10-06)

Compatible with the open kernel modules (Turing and newer), hourly prices excl. VAT:

Flavors GPU (architecture) Entry flavor
rtx5000-28/56/84 Quadro RTX 5000, 16 GB (Turing) rtx5000-28: 0.36 €/h
l4-90/180/360 L4, 24 GB (Ada) l4-90: 0.75 €/h
a10-45/90/180 A10, 24 GB (Ampere) a10-45: 0.76 €/h
l40s-90/180/360 L40S, 48 GB (Ada) l40s-90: 1.40 €/h
a100-180/360/720 A100, 80 GB (Ampere) a100-180: 2.75 €/h
h100-380/760/1520 H100, 80 GB (Hopper) h100-380: 2.80 €/h
h200-1920(-eph) 8× H200, 141 GB (Hopper) 42 €/h

Excluded (Volta, not supported by the open kernel modules): t1-*, t2-* (V100, V100S), including their -le variants.

With requirement-based selection, the cheapest match wins, e.g. 24 GB → l4-90 (then a10-45), 48 GB → l40s-90, 80 GB → a100-180 then h100-380 (barely more expensive, faster: a job may require Hopper explicitly). Gravelines has the widest choice, Paris fewer GPUs but H200s: a pool may fall back to another region, the node joining over the cross-region private network (higher latency and inter-region data transfer, fine for batch).

Open points

  1. NVIDIA driver: only GPUs supported by NVIDIA's open kernel modules are targeted, i.e. Turing and newer (T4, Ampere A10/A100, Ada L4/L40S, Hopper H100, Blackwell); older architectures (e.g. Volta V100/V100S, OVHcloud's t1/t2 flavors) are excluded until there is a need. The open modules build against recent kernels more easily than the proprietary ones, which avoids a dedicated LTS kernel for GPU images. Plus the container toolkit and the Kubernetes device plugin (or CDI). Pools only list compatible flavors.
  2. GPU model and flavors to target among the compatible ones (e.g. L4, L40S, A100, H100 at OVHcloud), availability per region/AZ, other providers' GPU offers (see providers.md).
  3. Data: batch jobs reading large datasets should read from S3 (or an attached Cinder volume), not copy everything to each new node.
  4. Image size and pull time (CUDA): bake the large images into the GPU node image, or use a registry cache in the cluster.