llm-d in Plain English: Serving LLMs on Kubernetes
I just ran a 72-billion-parameter model on EKS. Two GPUs, one router, a chat UI on top. The whole session cost about $6, and the GPUs were gone when I stopped.
This is what I learned - the parts that are actually different from serving a normal web app.
Why an LLM is not a web app
Three things break the usual Kubernetes playbook:
- The model has to fit in GPU memory. Weights live in VRAM, not RAM. Rule of thumb: about 1 byte per parameter at FP8, 2 at BF16. A 72B model in BF16 is roughly 145 GB. No single GPU holds that, so it gets split across two (tensor parallelism).
- GPU utilization lies. A GPU doing inference reads about 100% busy whether it is keeping up or hopelessly behind. CPU-style autoscaling cannot tell the difference.
- Requests have memory. While generating, the model keeps notes on the conversation in VRAM. That is the KV cache. Send the next message to a different pod and the notes are gone - it has to re-read the whole conversation.
Everything below is a consequence of those three.
Who decides what
Think of a busy restaurant. Customers queue at the door. A host decides which table each one gets. A manager decides how many waiters are on shift. A staffing agency decides how many kitchens to rent.
| Piece | In the restaurant | What it decides |
|---|---|---|
| llm-d router (EPP) | host | which pod gets this request |
| KEDA | manager | how many pods |
| Karpenter | staffing agency | how many GPU nodes |
| vLLM | the kitchen | actually runs the model |
Three of those sound alike. They are not. Mixing them up is the most common confusion.
What llm-d adds
A plain Kubernetes Service round-robins. For LLMs that wastes exactly what you paid for.
llm-d puts a router in front: Envoy plus an endpoint picker (EPP). The pool of model pods is described with the Gateway API Inference Extension's InferencePool API. For every request the router scores every pod:
- queue length - how busy is it right now
- KV-cache use - how full is its memory
- prefix-cache hit - does it already hold this conversation's notes
The same conversation lands on the pod that already has it. No re-reading. Faster first token, less wasted GPU. The scorers are described in the llm-d request scheduler docs and the llm-d post on intelligent inference scheduling.
Scale on queue depth, not GPU %
The router sees every request, so it knows how many are waiting. That is the real demand signal. KEDA reads it from Prometheus (the KEDA Prometheus scaler):
triggers:
- type: prometheus
metricType: AverageValue
metadata:
query: sum(llm_d_epp_flow_control_queue_size{model_name="..."})
threshold: "1" # one waiting request per replica = add a pod
Nothing waiting: you are keeping up. Something waiting: add capacity.
Here is the whole loop:
- Requests pile up. KEDA adds a pod.
- No free GPU, so the pod sits Pending.
- Karpenter sees it, reads what it asked for, and launches a matching GPU node.
- Traffic stops. Replicas drop. The node is empty.
- About a minute later, Karpenter gives it back (consolidation).
Idle GPU cost is zero. That is the whole point.
The VRAM math
+------------------ 2 x 96 GB GPU -------------------+
| weights ~135 GiB (split) | KV cache ~34 GiB |
+--------------------------------+--------------------+
fixed cost how many people fit
Weights are a fixed cost. What is left over is KV cache, and that is how many conversations fit at once. Squeeze it and requests queue. That is why queue depth is the right signal - it is a direct measure of running out of KV room.
For Kimi-Dev-72B on two 96 GB GPUs that is about 110,000 tokens of cache. Roughly three full 32k conversations at the same time.
What it looked like
Real numbers from the first run:
| Step | Time |
|---|---|
| GPU node launch | ~2 min |
| Download 145 GB of weights | ~6 min |
| Load + CUDA graph capture | ~4 min |
| Cold start, total | 12-15 min |
| Decode speed, one request | ~18 tokens/s |
The node costs about $8.40 an hour. A two-hour session lands around $17, and a quick proof of concept like mine around $6.
Running it: git is the deploy button
I split the work in two:
- Terraform + GitHub Actions build the AWS side: VPC, EKS, IAM, Karpenter, KEDA, Prometheus. The pipeline signs in to AWS with OIDC, so no keys are stored anywhere.
- Argo CD runs everything on the cluster from git: the router, the model server, the node pool, the autoscaler, the chat UI (Open WebUI).
Switching models is a one-line commit: tier: kimi. Setting it to none makes Argo CD prune the lot and Karpenter hand the GPUs back.
One catch worth knowing. With tier: none the app renders nothing, and Argo CD refuses to auto-sync a result that would delete every resource. You need allowEmpty: true on the root app or the shutdown quietly never happens. I only found it by testing the teardown.
The whole thing is open source: llm-d-eks-lab.
Sharp edges
What bit me, so it does not bite you:
- The default disk is slow. gp3 writes at 125 MB/s. That is about 20 minutes to land 145 GB. Raising it to 1000 MB/s costs around $0.05 an hour and cut the download to 6.
- Kubernetes gives a rollout 10 minutes.
progressDeadlineSecondsdefaults to 600. A big model is still loading, perfectly healthy, when the deploy gets marked failed. Derive the deadline from your startup probe. - The metric sits behind a feature gate. The EPP only emits queue size with
flowControlenabled. Without it the query returns nothing and nothing ever scales - with no error anywhere. - The GPU AMI has no device plugin. The EKS NVIDIA image ships drivers, not the NVIDIA device plugin DaemonSet. Without it nodes never advertise
nvidia.com/gpuand your pod waits on a node that is already billing. - Karpenter reserves
karpenter.sh/*tags. Put one in your node class and the whole object is rejected. No node pool, pod Pending forever. - vLLM rejects
tool_choice: "auto"until you pass--enable-auto-tool-choiceand a parser (vLLM tool calling docs). Chat UIs send it by default. - llm-d's wins need scale. Prefix-aware routing pays off across many replicas. With one or two you are mostly exercising the control plane. Fine for learning, not a benchmark.
If you are comparing serving layers, I covered the other big option in KServe in plain English.
The whole post in four lines
- An LLM needs its weights in VRAM, a KV cache per conversation, and a GPU that reads 100% busy either way.
- llm-d puts a router in front that picks the pod by queue, cache use and prefix hits.
- KEDA scales pods on queue depth. Karpenter buys GPU nodes for pending pods and returns them when empty.
- Keep it all in git and it costs about $8 an hour while you use it, nothing after.
The mental model that stuck: llm-d is a load balancer that knows what is in each GPU's memory.
References
llm-d
- llm-d project site and source on GitHub
- llm-d request scheduler docs - how the endpoint picker scores pods
- Intelligent inference scheduling with llm-d
- Kubernetes Gateway API Inference Extension - the project behind the
InferencePoolAPI the router uses
Engine and model
- vLLM on GitHub and the vLLM docs
- vLLM: optimization and tuning - tensor parallelism and KV cache sizing
- vLLM: tool calling -
--enable-auto-tool-choiceand the parsers - Efficient Memory Management for Large Language Model Serving with PagedAttention - the paper behind vLLM's KV cache handling
- Kimi-Dev-72B model card on Hugging Face, and Open WebUI
Scaling and cluster
- Kubernetes Deployments (the progress deadline)
- KEDA and its Prometheus scaler
- Karpenter, its NodeClasses and disruption (consolidation)
- Argo CD and its automated sync policy (
allowEmpty) - Envoy proxy
AWS
This project
- llm-d-eks-lab - all the Terraform, Argo CD apps and tiers behind this post
- KServe in plain English