A Digital Factory on Harvester: provisioning hundreds of RKE2 clusters on demand
Open-source hyperconvergence, virtual machines driven by KubeVirt, RKE2 clusters delivered in fifteen minutes. The complete architecture of a cluster factory, from the hardware foundation to the self-service catalogue.
The request comes up in every large organisation: teams want a Kubernetes cluster, right now, without raising a ticket that will take three weeks. The classic answer — a shared cluster with namespaces — does not fit every case: version trials, upgrade testing, regulated environments that require hardware isolation.
Harvester supplies the foundation that was missing. It is an open-source hyperconvergence solution built on Kubernetes, exposing virtual machines through KubeVirt and replicated storage through Longhorn. Combined with Rancher, it turns a fleet of servers into an RKE2 cluster factory.
The stack, from the bottom up
┌──────────────────────────────────────────────┐
│ Rancher — catalogue and self-service │
│ request → RKE2 cluster delivered in 15 min │
├──────────────────────────────────────────────┤
│ Harvester — hyperconvergence │
│ KubeVirt (compute) · Longhorn (storage) │
│ Multus + SR-IOV (network) │
├──────────────────────────────────────────────┤
│ Underlying RKE2 — Harvester platform │
├──────────────────────────────────────────────┤
│ Hardware — servers, NVMe, 25 GbE │
└──────────────────────────────────────────────┘The elegant part: Harvester is itself a Kubernetes cluster. Provisioning a virtual machine amounts to creating a custom resource. All the usual tooling applies — GitOps included.
Sizing the foundation
Three mistakes come back systematically on this kind of platform.
Undersizing the storage network. Longhorn replicates every volume three times, synchronously. A 100 MiB/s write generates 300 MiB/s of traffic. On 10 GbE shared with application traffic, saturation arrives quickly. Plan a dedicated 25 GbE storage network.
Mixing management traffic with virtual-machine traffic. An application traffic spike must never disturb cluster heartbeats.
Forgetting the recovery margin. If a node goes down, its virtual machines restart elsewhere. Without at least 25 % headroom, they stay pending.
| Role | Count | Configuration |
|---|---|---|
| Management nodes | 3 | 16 cores, 64 GiB, 2 × 480 GiB NVMe |
| Compute nodes | 12 to 40 | 64 cores, 512 GiB, 4 × 3.84 TiB NVMe |
| Management network | — | 2 × 10 GbE bonded |
| Machine network | — | 2 × 25 GbE, tagged VLANs |
| Storage network | — | 2 × 25 GbE dedicated to Longhorn |
Configuring the network
Harvester drives the network through ClusterNetwork and VlanConfig. Every tenant gets its VLAN.
apiVersion: network.harvesterhci.io/v1beta1
kind: ClusterNetwork
metadata:
name: vm-workload
---
apiVersion: network.harvesterhci.io/v1beta1
kind: VlanConfig
metadata:
name: vm-workload-uplink
spec:
clusterNetwork: vm-workload
nodeSelector:
harvesterhci.io/managed: "true"
uplink:
nics: ["ens5f0", "ens5f1"]
bondOptions: { mode: 802.3ad, miimon: 100 }
linkAttributes: { mtu: 9000 } # jumbo frames: +15 % usable throughput
---
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
name: vlan-team-paiement
namespace: harvester-public
labels:
network.harvesterhci.io/clusternetwork: vm-workload
network.harvesterhci.io/vlan-id: "412"
spec:
config: |
{
"cniVersion": "0.3.1",
"name": "vlan-team-paiement",
"type": "bridge",
"bridge": "vm-workload-br",
"promiscMode": true,
"vlan": 412,
"ipam": {}
}Jumbo frames at 9000 bytes are no detail: on replicated storage traffic they bring roughly 15 % extra usable throughput.
The virtual machine template
The key to fast provisioning lies in the image. A generic Ubuntu image spends eight minutes installing packages on every boot. An image prepared with Packer, already carrying the container engine and the RKE2 binaries, boots in ninety seconds.
apiVersion: harvesterhci.io/v1beta1
kind: VirtualMachineImage
metadata:
name: platform-rke2-1-31
namespace: harvester-public
spec:
displayName: "Socle RKE2 1.31 — durci"
sourceType: download
url: http://depot.interne/images/platform-rke2-1.31.4-durci.qcow2
checksum: "sha512:4f2a9c8e..."Hardening happens once, in the image, rather than on every boot. That is what makes the fifteen-minute promise keepable.
The self-service catalogue
Rancher exposes RKE2 provisioning on Harvester. A team states its need, the platform delivers.
apiVersion: provisioning.cattle.io/v1
kind: Cluster
metadata:
name: sandbox-paiement
namespace: fleet-default
spec:
kubernetesVersion: v1.31.4+rke2r1
rkeConfig:
machinePools:
- name: plan-controle
quantity: 3
etcdRole: true
controlPlaneRole: true
machineConfigRef:
kind: HarvesterConfig
name: gabarit-plan-controle
- name: travail
quantity: 3
workerRole: true
machineConfigRef:
kind: HarvesterConfig
name: gabarit-travail
machineGlobalConfig:
cni: calico
disable: ["rke2-ingress-nginx"] # the entry comes from the internal catalogue
etcd-snapshot-schedule-cron: "0 */6 * * *"
etcd-snapshot-retention: 20
kube-apiserver-arg:
- "audit-log-path=/var/log/audit/audit.log"
- "audit-policy-file=/etc/rancher/rke2/audit-policy.yaml"
upgradeStrategy:
controlPlaneConcurrency: "1"
workerConcurrency: "25%"
controlPlaneDrainOptions: { enabled: true, timeout: 600 }The machineGlobalConfig carries compliance: etcd backup every six hours, audit logging enabled, the default ingress controller disabled in favour of the internal catalogue one. Every cluster is born compliant.
Automating the request
Self-service without a guard produces three hundred orphan clusters in six months. An internal operator interposes an approval step and an expiry date.
// Reconciling a ClusterRequest: quota check, then creation.
func (r *ReconciliateurDemande) Reconcile(ctx context.Context,
req ctrl.Request) (ctrl.Result, error) {
var request platformev1.ClusterRequest
if err := r.Get(ctx, req.NamespacedName, &request); err != nil {
return ctrl.Result{}, client.IgnoreNotFound(err)
}
// Expiry: a sandbox cluster does not live forever.
if request.Spec.Duree != "" && request.Status.CreeLe != nil {
duration, _ := time.ParseDuration(request.Spec.Duree)
if time.Since(request.Status.CreeLe.Time) > duration {
r.Recorder.Event(&request, corev1.EventTypeNormal,
"Expiration", "durée de vie atteinte, suppression engagée")
return ctrl.Result{}, r.supprimerCluster(ctx, &request)
}
}
// Per-team quota: stops one team from absorbing the platform.
utilise, err := r.calculerQuotaEquipe(ctx, request.Spec.Equipe)
if err != nil {
return ctrl.Result{}, err
}
if utilise+request.Spec.Vcpu > r.quotaMax(request.Spec.Equipe) {
request.Status.Phase = "Refusee"
request.Status.Message = fmt.Sprintf(
"quota dépassé : %d vCPU utilisés, %d demandés, %d autorisés",
utilise, request.Spec.Vcpu, r.quotaMax(request.Spec.Equipe))
return ctrl.Result{}, r.Status().Update(ctx, &request)
}
return ctrl.Result{RequeueAfter: time.Hour}, r.creerCluster(ctx, &request)
}The expiry date is the most profitable mechanism on the whole platform. A sandbox cluster created for three days and still running eight months later is pure waste.
Holding up at scale
Past a hundred clusters, two limits appear.
The Harvester apiserver. Every virtual machine produces events and status updates. Three thousand machines saturate an undersized control plane. Watch apiserver_request_duration_seconds and plan to move etcd onto dedicated disks.
Longhorn rebuilds. Losing a node triggers the rebuild of all its replicas. Across 15 TiB, that means several hours of sustained traffic. Cap rebuild concurrency, or you turn a single node failure into a general degradation.
apiVersion: longhorn.io/v1beta2
kind: Setting
metadata:
name: concurrent-replica-rebuild-per-node-limit
namespace: longhorn-system
value: "2"The result
A team fills in a form. Fifteen minutes later it receives the access file for a six-node RKE2 cluster, backed up, audited, compliant with corporate policy, and scheduled to disappear on the announced date.
Operations no longer manage clusters but a platform. That is exactly the difference between absorbing demand and industrialising it.