Gardener: running a fleet of Kubernetes clusters without ever stretching one across two sites
One autonomous cluster per data centre, federated by a single control plane. How Gardener industrialises the lifecycle of dozens of clusters, and why the stretched cluster remains a bad idea.
Past a dozen clusters, the problem changes nature. It is no longer "how do we deploy a cluster" but "how do we guarantee that all forty clusters share the same version, the same policies and the same patch level".
Gardener, built by SAP and released as open source, answers that question with one strong idea: a Kubernetes cluster manages other Kubernetes clusters. The control plane of a managed cluster runs as an ordinary application workload.
The three-tier model
| Tier | Role | Count |
|---|---|---|
| Garden | Central control plane, catalogue, Shoot API | 1 |
| Seed | Hosts the control planes of managed clusters | 1 per site |
| Shoot | Working cluster handed to the teams | dozens |
The part that surprises at first: the apiserver and the etcd of a Shoot cluster run inside a Seed cluster, as pods. The Shoot holds nothing but its worker nodes.
Three practical consequences. Creating a cluster comes down to deploying a set of pods — minutes rather than an hour. Control-plane monitoring benefits from the Kubernetes tooling already in place. And upgrading etcd across forty clusters becomes a Helm chart upgrade.
Why never a stretched cluster
The idea of a single cluster spread over two data centres comes back regularly. It looks like high availability on the cheap. It does not hold.
etcd quorum does not survive a partition. With three members split two-one, losing the majority site puts the entire cluster into read-only. With five members split three-two, the minority site loses everything.
Latency degrades the control plane. etcd requires synchronous writes across the quorum. Past ten milliseconds between members, apiserver response times collapse — and with them everything that depends on it: scheduler, controllers, probes.
The blast radius doubles. A configuration mistake, a failed upgrade, an expired certificate: on a stretched cluster the incident hits both sites at once. That is exactly what the multi-site architecture set out to avoid.
The rule I apply. One autonomous cluster per data centre, with its own etcd and its own control plane. High availability gets built above the clusters — through routing, data replication and application failover — never inside a single one.
Declaring one Seed per site
apiVersion: core.gardener.cloud/v1beta1
kind: Seed
metadata:
name: seed-dc1-paris
spec:
provider:
type: openstack
region: paris-dc1
zones: ["az-a", "az-b", "az-c"] # three zones INSIDE the site, never across sites
networks:
nodes: 10.20.0.0/16
pods: 10.21.0.0/16
services: 10.22.0.0/17
shootDefaults:
pods: 100.96.0.0/11
services: 100.64.0.0/13
dns:
provider:
type: powerdns
secretRef: { name: dns-powerdns, namespace: garden }
ingress:
domain: ingress.dc1.interne.fr
controller: { kind: nginx }
settings:
scheduling: { visible: true }
verticalPodAutoscaler: { enabled: true }
topologyAwareRouting: { enabled: true } # traffic stays inside the zone
taints: []The three declared zones belong to the same site. That is the nuance that changes everything: you spread inside a data centre, not between data centres.
Declaring a working cluster
apiVersion: core.gardener.cloud/v1beta1
kind: Shoot
metadata:
name: production-paiement
namespace: garden-finance
spec:
cloudProfileName: openstack-interne
region: paris-dc1
seedSelector:
matchLabels: { site: dc1 }
secretBindingName: identifiants-finance
kubernetes:
version: "1.31.4"
kubeAPIServer:
auditConfig:
auditPolicy: { configMapRef: { name: politique-audit-stricte } }
oidcConfig:
issuerURL: https://sso.interne.fr/realms/company
clientID: kubernetes
usernameClaim: preferred_username
groupsClaim: groups
requests:
maxNonMutatingInflight: 600
maxMutatingInflight: 300
kubeControllerManager:
nodeMonitorGracePeriod: 40s
enableStaticTokenKubeconfig: false # OIDC authentication only
provider:
type: openstack
infrastructureConfig:
apiVersion: openstack.provider.extensions.gardener.cloud/v1alpha1
kind: InfrastructureConfig
networks: { workers: 10.180.0.0/16 }
workers:
- name: general
machine:
type: m1.xlarge
image: { name: gardenlinux, version: "1592.3.0" }
minimum: 3
maximum: 12
maxSurge: 2
zones: ["az-a", "az-b", "az-c"]
systemComponents: { allow: true }
- name: calcul
machine: { type: c2.4xlarge, image: { name: gardenlinux, version: "1592.3.0" } }
minimum: 0
maximum: 20
taints:
- { key: dedie, value: calcul, effect: NoSchedule }
zones: ["az-b"]
networking:
type: calico
pods: 100.96.0.0/11
services: 100.64.0.0/13
nodes: 10.180.0.0/16
maintenance:
timeWindow: { begin: "020000+0100", end: "040000+0100" }
autoUpdate:
kubernetesVersion: false # never automatic in production
machineImageVersion: true
hibernation:
enabled: false
schedules:
- start: "0 20 * * 1-5" # on non-production environments
end: "0 7 * * 1-5"
location: "Europe/Paris"Three items deserve a comment.
enableStaticTokenKubeconfig: false removes the static-token access file. Authentication goes exclusively through OIDC, therefore through your directory — immediate revocation included.
autoUpdate.kubernetesVersion: false on production. Gardener knows how to upgrade on its own; let it do so on staging, never on production without prior validation.
Scheduled hibernation switches non-production clusters off overnight and at weekends. Across twenty development clusters, the saving reaches 60 % of the compute bill.
High availability, built above
Every site carries its autonomous cluster. Continuity gets built one storey up.
┌──────────────────────┐
Users ─▶│ Global balancer │ health check every 5 s
└───────┬──────────────┘
┌───────────┴───────────┐
▼ ▼
┌─────────────────────┐ ┌─────────────────────┐
│ Shoot dc1-paris │ │ Shoot dc2-lyon │
│ its own etcd │ │ its own etcd │
│ 3 internal zones │ │ 3 internal zones │
└──────────┬──────────┘ └──────────┬──────────┘
└────── asynchronous ────┘
data replicationThe global balancer moves traffic when a site stops answering. Data replicates asynchronously between the two sites — PostgreSQL logical replication, or replicated object storage. You accept losing a few seconds of transactions rather than risking a complete outage.
Policies, applied everywhere
The point of a managed fleet is uniformity. Gardener applies extensions to every Shoot matching a selector.
apiVersion: core.gardener.cloud/v1beta1
kind: ControllerRegistration
metadata:
name: extension-securite-interne
spec:
resources:
- { kind: Extension, type: securite-interne, globallyEnabled: true }
deployment:
policy: Always
deploymentRefs: [{ name: extension-securite-interne }]The extension installs Kyverno, the corporate policies, the log-collection agent and the etcd backup trigger. A new cluster is born compliant, without the team requesting it having to think about it.
What Gardener does not solve
Network design stays yours. Gardener provisions; it does not decide your addressing plan or your interconnection rules.
The cost of a Seed. Every Seed is a full Kubernetes cluster to size and to operate. Below ten Shoots per site, the economics stay debatable.
The learning curve. The three-tier model disorients. Allow two to three weeks before the team naturally thinks in Garden, Seed and Shoot.
Past fifteen clusters, the investment pays for itself without argument. Below that, a well-kept Terraform repository often does the job.