vk-cocoon: running Kubernetes pods as MicroVMs rather than containers

A Linux container shares its host kernel. That is what makes it light — and it is also its security limit. One escape flaw in the kernel compromises everything running on the machine.

For most internal workloads, that risk stays acceptable. It stops being acceptable the moment you run code whose origin you do not control: a third-party customer workload, a function supplied by a partner, edge processing on a physically accessible site.

The vk-cocoon project offers an elegant answer. Rather than replacing the container runtime, it presents itself to the cluster as a virtual node. The pods assigned to it do not become containers but MicroVMs, each with its own kernel.

How the Virtual Kubelet mechanism works

Virtual Kubelet is a CNCF project that simulates a kubelet. It registers a Node with the cluster, the scheduler assigns pods to it, and the provider behind it does whatever it likes with those pods.

   kube-scheduler
         │ assigns the pod to node “cocoon-edge-01”
         ▼
   ┌─────────────────────────┐
   │  vk-cocoon              │  registers as a Node
   │  (Virtual Kubelet)      │  translates PodSpec → MicroVM
   └────────────┬────────────┘
                ▼
   ┌─────────────────────────┐
   │  Cloud Hypervisor       │  one kernel per workload
   │  or Firecracker         │  boot ~125 ms
   └─────────────────────────┘

The value lies in what does not change. Your manifests stay Kubernetes manifests. kubectl logs works, kubectl exec too. Probes, secrets, ConfigMaps: everything behaves normally. Only the execution mode differs.

Installation

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: vk-cocoon
  namespace: cocoon-system
spec:
  selector:
    matchLabels: { app: vk-cocoon }
  template:
    metadata:
      labels: { app: vk-cocoon }
    spec:
      nodeSelector:
        cocoon.io/hypervisor: "enabled"    # nodes with /dev/kvm available
      hostNetwork: true
      serviceAccountName: vk-cocoon
      containers:
        - name: provider
          image: ghcr.io/cocoonstack/vk-cocoon:v0.9.3
          args:
            - --provider=cocoon
            - --nodename=cocoon-$(NOM_HOTE)
            - --hypervisor=cloud-hypervisor
            - --kernel-image=/var/lib/cocoon/vmlinux-6.12-microvm
            - --rootfs-template=/var/lib/cocoon/rootfs-base.img
            - --vm-boot-timeout=20s
            - --klog.v=2
          env:
            - name: NOM_HOTE
              valueFrom: { fieldRef: { fieldPath: spec.nodeName } }
            - name: KUBELET_PORT
              value: "10260"
          securityContext:
            privileged: true                # access to /dev/kvm and to tap creation
          volumeMounts:
            - { name: kvm,     mountPath: /dev/kvm }
            - { name: cocoon,  mountPath: /var/lib/cocoon }
            - { name: run,     mountPath: /run/cocoon }
          resources:
            requests: { cpu: 500m, memory: 512Mi }
      volumes:
        - name: kvm
          hostPath: { path: /dev/kvm, type: CharDevice }
        - name: cocoon
          hostPath: { path: /var/lib/cocoon, type: DirectoryOrCreate }
        - name: run
          hostPath: { path: /run/cocoon, type: DirectoryOrCreate }

The virtual node then appears with a taint that stops the scheduler placing just anything on it.

kubectl get nodes -l type=virtual-kubelet
# NAME                STATUS   ROLES   VERSION
# cocoon-edge-01      Ready    agent   v1.31.4-vk-cocoon

kubectl describe node cocoon-edge-01 | grep -A2 Taints
# Taints: cocoon.io/microvm=true:NoSchedule

Assigning a workload

Only pods that explicitly tolerate the taint land as MicroVMs.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: traitement-partenaire
  namespace: payloads-externes
spec:
  replicas: 4
  selector:
    matchLabels: { app: traitement-partenaire }
  template:
    metadata:
      labels: { app: traitement-partenaire }
      annotations:
        cocoon.io/vcpus: "2"
        cocoon.io/memory: "2048"
        cocoon.io/kernel-cmdline: "quiet reboot=k panic=1 pci=off"
        cocoon.io/balloon: "true"          # reclaim of unused memory
    spec:
      nodeSelector: { type: virtual-kubelet }
      tolerations:
        - key: cocoon.io/microvm
          operator: Equal
          value: "true"
          effect: NoSchedule
      containers:
        - name: traitement
          image: registry-partenaire.io/traitement:4.1.0
          resources:
            requests: { cpu: "2", memory: 2Gi }
            limits:   { cpu: "2", memory: 2Gi }   # equal: the MicroVM is fixed
          ports: [{ containerPort: 8080 }]

Note the equality between requests and limits. A MicroVM receives a fixed allocation at boot; the notion of controlled overcommit does not exist as it does for a cgroup. The memory balloon lets unused memory be returned, but the ceiling stays firm.

What isolation really costs

Those figures cut against the image of heavyweight virtualisation. Cloud Hypervisor boots a MicroVM in a hundred and twenty milliseconds because it emulates almost nothing: no BIOS, no PCI, just virtio and a kernel cut for the purpose.

Memory overhead stays the most tangible point. Forty megabytes per pod matters when you run five hundred of them. On an external workload consuming two gigabytes, it is noise.

The limits to know about

No shared kernel, therefore no shared kernel features. A pod needing hostPID, hostNetwork or privileged capabilities will not work. That is the point, but it rules out monitoring agents that inspect the host.

Persistent volumes go through virtio-fs or virtio-blk. Performance is respectable, but an in-memory emptyDir behaves differently. Test your write-intensive workloads.

The service mesh needs attention. An Istio sidecar runs inside the same MicroVM as the application — which stays consistent. But automatic injection must be verified; some providers do not handle it.

The project is still young. Version 0.9 is not a 1.0. Test seriously before putting a critical workload on it.

When it is worth the detour

Three situations justify the investment.

Running third-party code. A platform running functions supplied by customers cannot settle for a shared kernel.

Physically exposed environments. An edge server, in a factory or a plant room, that someone can plug a cable into.

Regulatory isolation requirements. Some frameworks require hypervisor-level separation between processing of different sensitivity levels. Cocoon provides it without leaving Kubernetes.

For everything else, a well-configured container — unprivileged user, read-only file system, dropped capabilities, seccomp profile — covers most of the risk at zero cost.