KServe: serving a bespoke model that watches its own cluster
Scale to zero, canary rollout, built-in drift capture: KServe brings to the inference service what Knative brings to applications. A complete implementation on a home-grown model.
Kubeflow orchestrates training. What remains is serving the model — and that is where most MLOps chains come apart. A Deployment with a Flask API works in a demo; it holds production badly.
KServe answers three needs classic deployment does not cover: scaling to zero when the model serves nobody, progressive rollout between versions, and request capture for drift detection.
What KServe genuinely brings
The InferenceService manifest describes a complete inference service. Behind it, KServe assembles Knative Serving for elasticity, Istio for routing, and a runtime matched to the model format.
| Need | Without KServe | With KServe |
|---|---|---|
| Scale to zero | Impossible with a Deployment | Native, through Knative |
| Canary between versions | Two services and a manual switch | canaryTrafficPercent |
| Pre- and post-processing | Inside the model code | A separate transformer |
| Request capture | To be written | Built-in logger |
| Model format | Server to be written | Runtimes provided |
The split between transformer and predictor is worth dwelling on: it lets data preparation evolve without touching the model, and the other way round.
The custom runtime
For a standard scikit-learn model, the provided runtime is enough. Our anomaly detector asks for more: it has to query Prometheus, build the input vector and normalise before predicting.
"""KServe inference server for the cluster anomaly detector."""
import argparse
from typing import Any
import joblib
import kserve
import numpy as np
from prometheus_api_client import PrometheusConnect
FEATURES = ["cpu_utilisation", "memory_bytes", "restarts",
"p99_latency", "error_rate", "scheduler_wait"]
class AnomalyDetector(kserve.Model):
"""Predicts an anomaly score from a pod identifier."""
def __init__(self, nom: str, model_path: str, url_prometheus: str):
super().__init__(nom)
self.path = model_path
self.prometheus = PrometheusConnect(url=url_prometheus, disable_ssl=False)
self.model = None
self.scaler = None
self.ready = False
def load(self) -> bool:
"""Loaded once, when the container starts."""
artifact = joblib.load(f"{self.path}/model.joblib")
self.model = artifact["detector"]
self.scaler = artifact["scaler"]
self.ready = True
return self.ready
def preprocess(self, payload: dict[str, Any], headers=None) -> dict[str, Any]:
"""Fetches the current pod state from Prometheus."""
vectors = []
for entry in payload["instances"]:
pod = entry["pod"]
measures = []
for variable in FEATURES:
query = f'{variable}{{pod="{pod}"}}'
result = self.prometheus.custom_query(query=query)
measures.append(float(result[0]["value"][1]) if result else 0.0)
vectors.append(measures)
return {"vectors": vectors, "pods": [e["pod"] for e in payload["instances"]]}
def predict(self, inputs: dict[str, Any], headers=None) -> dict[str, Any]:
X = self.scaler.transform(np.array(inputs["vectors"]))
scores = self.model.decision_function(X)
verdicts = self.model.predict(X) # -1 anomaly, 1 normal
return {"predictions": [
{
"pod": pod,
"score": round(float(s), 4),
"anomaly": bool(v == -1),
# The raw Isolation Forest score is not a probability:
# we map it into [0,1] so operations can read it.
"confiance": round(float(min(1.0, abs(s) * 2)), 3),
}
for pod, s, v in zip(inputs["pods"], scores, verdicts)
]}
if __name__ == "__main__":
analyseur = argparse.ArgumentParser(parents=[kserve.model_server.parser])
analyseur.add_argument("--model_dir", required=True)
analyseur.add_argument("--prometheus_url", required=True)
args, _ = analyseur.parse_known_args()
model = AnomalyDetector("detector-anomalies", args.model_dir, args.prometheus_url)
model.load()
kserve.ModelServer().start([model])The comment about the score deserves attention. Isolation Forest decision_function returns a signed distance, not a probability. Exposing it as it stands leads operations teams to read it as a percentage — and to get it wrong.
The inference manifest
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
name: detector-anomalies
namespace: mlops
annotations:
serving.kserve.io/deploymentMode: Serverless
spec:
predictor:
minReplicas: 1 # no scale to zero: loading the model costs 20 s
maxReplicas: 8
scaleTarget: 30 # concurrent requests per replica
scaleMetric: concurrency
containerConcurrency: 30
serviceAccountName: kserve-mlops
logger:
mode: all # captures inputs and outputs for drift
url: http://collecteur-derive.mlops.svc/log
containers:
- name: kserve-container
image: registry.interne/detector-anomalies:2.3.0
args:
- --model_dir=/mnt/models/detector
- --prometheus_url=http://prometheus.observabilite.svc:9090
resources:
requests: { cpu: 500m, memory: 1Gi }
limits: { cpu: "2", memory: 3Gi }
readinessProbe:
httpGet: { path: /v1/models/detector-anomalies, port: 8080 }
initialDelaySeconds: 25
volumes:
- name: models
persistentVolumeClaim: { claimName: registre-models }Two choices deserve justification.
minReplicas: 1 rather than zero. Scaling to zero is appealing on paper. In practice, the first call after a quiet spell pays for the container start plus the model load — twenty seconds here. For a detection service called every minute, the saving does not offset the latency. Keep zero for genuinely sporadic models.
scaleMetric: concurrency rather than rps. An inference request that queries Prometheus spends most of its time waiting. Measuring requests per second underestimates the real load; measuring concurrency reflects the number of requests in flight.
The canary rollout
This is KServe strongest argument. Running two versions side by side and shifting traffic gradually fits in one line.
spec:
predictor:
canaryTrafficPercent: 10
containers:
- name: kserve-container
image: registry.interne/detector-anomalies:2.4.0 # nouvelle versionTen per cent of the traffic goes to 2.4.0, the rest stays on 2.3.0. Both versions expose their metrics separately, which lets you compare before switching over.
# Comparing both revisions on the reported anomaly rate
sum by (revision) (
rate(kserve_anomalies_detected_total{service="detector-anomalies"}[15m])
)
/
sum by (revision) (
rate(kserve_predictions_total{service="detector-anomalies"}[15m])
)If the new version reports three times as many anomalies as the old one on the same traffic, the problem lies with the model, not the cluster. You set canaryTrafficPercent back to zero without having broken anything.
Drift detection wired in the right place
The KServe logger sends every input-output pair to a collector. That is what feeds drift analysis — without instrumenting the model code.
apiVersion: apps/v1
kind: Deployment
metadata:
name: collecteur-derive
namespace: mlops
spec:
replicas: 2
selector:
matchLabels: { app: collecteur-derive }
template:
metadata:
labels: { app: collecteur-derive }
spec:
containers:
- name: collecteur
image: registry.interne/collecteur-derive:1.2.0
env:
- name: FENETRE_REFERENCE
value: "7d" # rolling seven-day comparison
- name: SEUIL_ALERTE
value: "0.3"
ports:
- { name: http, containerPort: 8080 }
- { name: metriques, containerPort: 9090 }The collector accumulates requests, compares the current distribution with the previous seven days as a reference, and publishes the result as Prometheus metrics. The alert then gets configured exactly like any infrastructure alert.
KServe or Kubeflow: the question does not arise
The two complement each other. Kubeflow Pipelines trains and publishes to the MLflow registry; KServe serves the version carrying the production alias. The link is a simple annotation.
metadata:
annotations:
mlflow.org/model-uri: "models:/detector-anomalies-cluster@production"A home-grown operator watches the alias in the registry and updates the InferenceService image when it changes. Monday morning retraining thus propagates all the way to the inference service with no intervention.
The mistakes I have seen
Loading the model on every request. load() exists to be called once. Putting it inside predict() multiplies latency by fifty.
Forgetting the readiness probe. Without it, Knative routes traffic to a pod whose model has yet to load. The first call fails, and Knative considers the pod healthy.
Sizing memory to the model file. A 200 MiB model loads into three or four times that once deserialised, before counting the request buffer. Measure under load, do not calculate.
Treating drift as a research subject. It gets watched with the same tools as everything else: a metric, a threshold, an alert. A dedicated dashboard nobody looks at serves no purpose.