On-premise and hybrid AI infrastructure: building a Kubernetes foundation that holds GPU load

The question has stopped being a matter of fashion. Three constraints are pushing engineering leadership to bring part of their artificial-intelligence workloads back onto their own infrastructure: data confidentiality, the recurring cost of inference billed by the token, and the latency of outbound calls.

That said, deploying a model behind your firewall differs deeply from calling an API. AI workloads break the assumptions most Kubernetes clusters rest on: they demand indivisible resources, move considerable volumes of data, and cope badly with preemption.

What changes compared with a classic application workload

An application container divides finely. You allocate it 250 millicores and 512 MiB, the scheduler places it wherever room exists, and you overcommit without much risk.

A GPU does not divide that way. nvidia.com/gpu remains a whole resource, not overcommittable by default. Three immediate consequences follow.

Allocation granularity turns coarse. A pod that claims a GPU monopolises it, even when it uses only 12 % of it. On a card costing 25 000 euros, the waste adds up fast.

Start-up time explodes. An image carrying CUDA, PyTorch and 30 GiB of model weights does not download in two seconds. Without a pre-loading strategy, you wait several minutes before the first token.

Scheduling becomes a batch problem. Training distributed over eight GPUs only starts if all eight are free at the same moment. The default scheduler, which places pods one by one, causes deadlocks: four pods reserve four GPUs and wait indefinitely for the other four.

The foundation: GPU operator and container runtime

Everything starts with exposing the hardware properly. The NVIDIA GPU operator installs and maintains the whole chain — driver, container runtime, device plugin, metrics exporter — without manual work on the nodes.

# GPU operator values, through Helm and managed by Argo CD
driver:
  enabled: true
  version: "580.65.06"
  rdma:
    enabled: true          # required for multi-node training
toolkit:
  enabled: true
devicePlugin:
  config:
    name: config-gpu
    default: partage-temporel
migManager:
  enabled: true            # hardware partitioning on A100, H100, H200
dcgmExporter:
  enabled: true
  serviceMonitor:
    enabled: true
nodeStatusExporter:
  enabled: true

The structuring point sits in devicePlugin.config. That is what decides how you share one card between several workloads.

Three sharing strategies, three uses

In multi-tenant production, MIG is the only defensible choice. A 1g.10gb profile guarantees each tenant its memory and its compute units. On an 80 GiB H100 you get seven independent instances — and seven inference workloads that never tread on each other.

# MIG layout declared by node label, applied by the mig-manager
apiVersion: v1
kind: ConfigMap
metadata:
  name: config-mig-personnalisee
  namespace: gpu-operator
data:
  config.yaml: |
    version: v1
    mig-configs:
      inference-dense:
        - devices: [0, 1, 2, 3]
          mig-enabled: true
          mig-devices:
            "1g.10gb": 7
      training:
        - devices: [4, 5, 6, 7]
          mig-enabled: false

The nvidia.com/mig.config=inference-dense label flips the node. The mig-manager drains, reconfigures and brings it back — an operation that stays disruptive, so schedule it outside peak hours.

Scheduling: dropping the default scheduler

For distributed training, the native scheduler will not do. It lacks gang scheduling: either every pod in a job gets its resources, or none of them starts.

Two proven options: Volcano and Kueue. Kueue has the advantage of plugging natively into Job objects and resource queues, without introducing a parallel pod type.

apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: file-research
spec:
  namespaceSelector: {}
  cohort: gpu-partage
  resourceGroups:
    - coveredResources: ["cpu", "memory", "nvidia.com/gpu"]
      flavors:
        - name: h100-sxm
          resources:
            - name: "nvidia.com/gpu"
              nominalQuota: 16
              borrowingLimit: 8      # borrowing from the other queues in the cohort
            - name: "cpu"
              nominalQuota: 512
            - name: "memory"
              nominalQuota: 4Ti
  preemption:
    reclaimWithinCohort: Any
    withinClusterQueue: LowerPriority

The cohort mechanism solves the political problem as much as the technical one: the research team holds sixteen guaranteed GPUs, can borrow eight from production while production idles, and returns them the moment production asks. That avoids the rigid partition which leaves expensive cards asleep.

Storage, the blind spot of most projects

Architectures rarely fail on compute. They fail on data access.

A training run reading a 4 TiB dataset through NFS saturates the network and leaves the GPUs idle. The symptom shows directly in the metrics: DCGM_FI_DEV_GPU_UTIL plateaus at 30 % while the network card sits saturated.

Three layers to keep apart:

Object storage holds raw datasets and model weights. MinIO or Ceph RGW on site. S3 access, versioning, replication.

Local NVMe cache feeds training. A local-path-provisioner pointing at NVMe SSDs, with the dataset pre-loaded before the job starts. Throughput goes from 1 GiB/s over NFS to 12 GiB/s locally.

Shared high-performance storage covers the cases where the dataset outgrows the local disk. CephFS through Rook, or a parallel file system such as Lustre for the most demanding workloads.

# Dataset pre-loading before training,
# run as an init container
initContainers:
  - name: hydrater-cache
    image: registry.interne/mc:RELEASE.2026-08-14
    command:
      - sh
      - -c
      - |
        set -euo pipefail
        mc alias set depot http://minio.stockage.svc:9000 "$CLE" "$SECRET"
        mc mirror --overwrite --retry \
          depot/jeux-data/vision-2026/ /cache/vision-2026/
        echo "Cache warmed: $(du -sh /cache/vision-2026 | cut -f1)"
    volumeMounts:
      - name: cache-nvme
        mountPath: /cache

Networking: RDMA stops being optional past four nodes

In distributed training, gradient synchronisation dominates cycle time. Over classic 25 Gbit/s Ethernet, that phase consumes up to 60 % of the total. The GPUs wait for the network.

The answer comes in two pieces: RoCEv2 or InfiniBand on the hardware side, and the NVIDIA network operator on the Kubernetes side, which exposes RDMA interfaces to pods through secondary devices.

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: rdma-training
  namespace: ia-production
spec:
  config: |
    {
      "cniVersion": "0.3.1",
      "type": "sriov",
      "name": "rdma-training",
      "ipam": { "type": "whereabouts", "range": "10.56.0.0/16" }
    }

The pod then declares the annotation k8s.v1.cni.cncf.io/networks: rdma-training and claims the rdma/hca_shared_devices_a resource. NCCL detects the transport automatically and switches over.

Measure before you invest. An all_reduce_perf run from nccl-tests on your real topology tells you in ten minutes whether the network is your bottleneck. On eight H100-equipped nodes, expect more than 300 GiB/s of aggregate bandwidth. Below 50, something is wrong.

Hybrid: which workload stays where

A purely local architecture hits one wall: peaks. You do not size a fleet to absorb one quarter's spike. Hybrid answers that elasticity.

The split follows two axes — data sensitivity and load variability.

Technically, two separate clusters federated through GitOps give far better results than one cluster stretched between site and cloud. Latency between control planes makes the stretched option fragile: one link outage and your etcd loses quorum.

dépôt-gitops/
├── platform/
│   ├── gpu-operator/
│   ├── kueue/
│   └── observabilite/
├── clusters/
│   ├── onprem-dc1/          # H100, sensitive data
│   │   └── kustomization.yaml
│   ├── onprem-dc2/          # A100, recette
│   └── cloud-burst/         # provisioned on demand
└── payloads/
    ├── inference-interne/   → onprem-dc1 uniquement
    ├── training/        → onprem-dc1, onprem-dc2
    └── research/           → cloud-burst

Each cluster stays autonomous, with its own etcd and its own control plane. Argo CD, deployed on a management cluster, reconciles all three. A network outage isolates a cluster without breaking it.

Observability: three signals that genuinely matter

GPU dashboards overflow with useless metrics. Three are enough to catch what counts.

Effective compute usage — DCGM_FI_PROF_SM_ACTIVE — rather than GPU_UTIL, which only measures whether a kernel exists. A card can report 100 % utilisation with 4 % real multiprocessor activity.

Allocated memory against available memory — DCGM_FI_DEV_FB_USED. An out-of-memory kills the process without ceremony, and the training logs go with it.

Scheduling wait time — exposed by Kueue through kueue_pending_workloads. If your jobs queue for three hours, your problem is not compute power but allocation policy.

# Cards reserved but genuinely idle — the waste, in euros
(
  count by (node) (DCGM_FI_DEV_FB_USED > 0)
  -
  count by (node) (DCGM_FI_PROF_SM_ACTIVE > 0.15)
) > 0

That query often turns up surprises: forgotten notebooks holding a card hostage for twelve days.

The real cost, in figures

A server with eight H100 SXM costs around 280 000 euros, amortised over four years, which is roughly 5 800 euros a month. Add electricity — 10 kW under load, so 1 100 euros a month at 0.15 euro per kilowatt-hour — plus cooling and hosting: you approach 8 000 euros monthly for eight GPUs.

The same capacity at a cloud provider bills between 25 000 and 45 000 euros a month on a committed reservation. The equation tips towards on-site as soon as utilisation passes 35 to 40 %.

Below that, the cloud stays cheaper. And that is precisely where MIG sharing and a resource-queue policy make the difference: they turn a fleet running at 25 % into a fleet running at 70 %.

What the field has taught me

Start by measuring your real utilisation before buying anything. Most organisations discover their existing GPUs are asleep.

Treat storage and networking as first-class subjects, not as housekeeping details. A compute cluster starved of data costs as much as a well-designed one, for half the result.

Separate clusters rather than stretching them. GitOps federation brings consistency without the fragility.

Finally, automate MIG reconfiguration and resource queues from day one. Whatever stays manual always drifts in the end — and on resources this expensive, drift gets paid in cash.