Kairos: treating the operating system as a container image

Managing a hundred servers in a data centre is a solved problem. Managing a hundred servers spread across a hundred sites, with nobody on site and sometimes behind an unstable link, is not.

Classic configuration tools assume a mutable system that you converge. On an edge fleet, that assumption turns against you: after two years, every machine has drifted differently, and nobody knows exactly what runs where.

Kairos reverses the logic. The operating system becomes a container image: built by a CI pipeline, versioned in a registry, distributed and booted as it stands. The machine no longer gets configured, it gets replaced.

The principle

# The operating system is described in a Dockerfile
FROM quay.io/kairos/ubuntu:24.04-standard-amd64-generic-v3.6.0-k3sv1.31.4-k3s1

# Site-specific packages
RUN apt-get update && apt-get install -y --no-install-recommends       chrony lldpd smartmontools nvme-cli     && rm -rf /var/lib/apt/lists/*

# Internal monitoring agent
COPY --chmod=0755 agent-supervision /usr/local/bin/agent-supervision
COPY agent-supervision.service /etc/systemd/system/
RUN systemctl enable agent-supervision.service

# Hardening applied at build time, not at boot
COPY durcissement/ /etc/
RUN /usr/local/bin/appliquer-durcissement.sh

# Internal certificate authority
COPY pki/ca-interne.crt /usr/local/share/ca-certificates/
RUN update-ca-certificates

That image builds in GitLab CI, gets signed, and is pushed to the internal registry. Every machine in the fleet boots exactly the same one — bit for bit.

The most useful consequence. "Which version runs at the Lyon site?" becomes a question with a single answer: the image digest. Drift stops being possible.

Installation with no one on site

Kairos boots from an ISO image, a USB stick or the network, then reads its configuration from a cloud-init style file.

#cloud-config

install:
  device: "auto"
  auto: true
  reboot: true
  image: "oci:registry.interne/edge/platform-k3s:v3.6.0-r12"
  grub_options:
    extra_cmdline: "rd.neednet=1 console=tty1 console=ttyS0,115200"

users:
  - name: exploitation
    groups: ["admin"]
    ssh_authorized_keys:
      - "ssh-ed25519 AAAAC3Nza... exploitation@pi-groupe"
    lockPasswd: true          # no password login

k3s:
  enabled: true
  args:
    - --disable=traefik
    - --disable=servicelb
    - --flannel-backend=wireguard-native     # node-to-node encryption
    - --node-label=site=lyon-usine-2
    - --node-label=topologie=edge
    - --kubelet-arg=eviction-hard=memory.available<200Mi
    - --tls-san=k3s.lyon-usine-2.interne

kairos:
  network_token: "{{ p2p_network_token }}"    # encrypted mesh between site nodes

stages:
  boot:
    - name: "Attendre l'clock"
      commands:
        - chronyc waitsync 30 0.1 || true     # certificates fail without accurate time
  network:
    - name: "Register the machine in the inventory"
      commands:
        - |
          curl -sf --retry 5 --retry-delay 10             -H "Authorization: Bearer ${JETON_INVENTAIRE}"             -X POST https://inventaire.interne/api/nodes             -d "{\"serie\":\"$(dmidecode -s system-serial-number)\",\"site\":\"lyon-usine-2\"}"

The clock-synchronisation step is not cosmetic. An edge server whose motherboard battery has died boots in 1970. Every TLS certificate is then invalid, and the node never joins the cluster. That failure accounts for a surprising share of installation incidents.

The atomic upgrade

This is Kairos decisive contribution. Two system partitions coexist: the active one and the passive one. An update writes into the passive one, flips the boot loader, reboots. On failure, the rollback happens by itself.

apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
  name: montee-platform-v3-7
  namespace: system-upgrade
spec:
  concurrency: 2                    # two machines at a time across the whole fleet
  nodeSelector:
    matchExpressions:
      - { key: topologie, operator: In, values: ["edge"] }
      - { key: canal-maj, operator: In, values: ["stable"] }
  serviceAccountName: system-upgrade
  cordon: true
  drain:
    force: true
    skipWaitForDeleteTimeout: 60
  upgrade:
    image: quay.io/kairos/kairos-upgrade
    args:
      - "--source=oci:registry.interne/edge/platform-k3s:v3.7.0-r3"
      - "--verify-signature"
    envs:
      - name: COSIGN_PUBLIC_KEY
        valueFrom: { secretKeyRef: { name: key-signature, key: cosign.pub } }

Signature verification before the switch is essential. Without it, whoever can push to the registry decides which system boots on your sites.

Progressiveness matters just as much. concurrency: 2 across a fleet of eighty machines spreads the upgrade over several hours. A regression shows up on two sites, not on eighty.

The rollout channel

Progressive rollout by label reuses the proven pattern of fleet updates.

kubectl label node edge-lyon-01 edge-nantes-03 canal-maj=canary --overwrite

Twenty-one days between the canary and the critical sites leave time to discover what no test reveals: incompatibility with an industrial controller, a network driver that regresses, a card that dislikes the new kernel.

Immutability in practice

The root file system mounts read-only. Whatever you need to keep gets declared explicitly.

install:
  bind_mounts:
    - /var/lib/rancher
    - /var/lib/kubelet
    - /var/lib/data-metier
  ephemeral_mounts:
    - /tmp
    - /var/tmp

That constraint forces a healthy discipline. If a workload writes to an undeclared path, it loses its data on reboot — and you find out in staging rather than in production. The system documents what matters.

What you have to accept

Troubleshooting changes nature. You cannot install a package to diagnose. Plan your tools into the image, or a ready-made debug container.

Image weight counts. A base with K3s runs around 1.2 GiB. On a 4 Mbit/s site, the upgrade takes forty minutes. Mind your layers, or plan a local mirror.

The build becomes a critical component. Your CI pipeline now produces the operating system of your fleet. It deserves the same level of protection as production.

The verdict

The gain is not technical, it is organisational. An edge fleet on Kairos gets reasoned about like an application rollout: one version, one channel, a progressive deployment, a rollback.

The questions that used to poison operations — which version, which patch, which configuration — get one verifiable answer. Across a hundred sites, that changes everything.