llm-d in Plain English: Serving LLMs on Kubernetes

I just ran a 72-billion-parameter model on EKS. Two GPUs, one router, a chat UI on top. The whole session cost about $6, and the GPUs were gone when I stopped.

This is what I learned - the parts that are actually different from serving a normal web app.

Why an LLM is not a web app

Three things break the usual Kubernetes playbook:

  1. The model has to fit in GPU memory. Weights live in VRAM, not RAM. Rule of thumb: about 1 byte per parameter at FP8, 2 at BF16. A 72B model in BF16 is roughly 145 GB. No single GPU holds that, so it gets split across two (tensor parallelism).
  2. GPU utilization lies. A GPU doing inference reads about 100% busy whether it is keeping up or hopelessly behind. CPU-style autoscaling cannot tell the difference.
  3. Requests have memory. While generating, the model keeps notes on the conversation in VRAM. That is the KV cache. Send the next message to a different pod and the notes are gone - it has to re-read the whole conversation.

Everything below is a consequence of those three.

Who decides what

Think of a busy restaurant. Customers queue at the door. A host decides which table each one gets. A manager decides how many waiters are on shift. A staffing agency decides how many kitchens to rent.

Piece In the restaurant What it decides
llm-d router (EPP) host which pod gets this request
KEDA manager how many pods
Karpenter staffing agency how many GPU nodes
vLLM the kitchen actually runs the model

Three of those sound alike. They are not. Mixing them up is the most common confusion.

What llm-d adds

A plain Kubernetes Service round-robins. For LLMs that wastes exactly what you paid for.

llm-d puts a router in front: Envoy plus an endpoint picker (EPP). The pool of model pods is described with the Gateway API Inference Extension's InferencePool API. For every request the router scores every pod:

  • queue length - how busy is it right now
  • KV-cache use - how full is its memory
  • prefix-cache hit - does it already hold this conversation's notes

The same conversation lands on the pod that already has it. No re-reading. Faster first token, less wasted GPU. The scorers are described in the llm-d request scheduler docs and the llm-d post on intelligent inference scheduling.

Scale on queue depth, not GPU %

The router sees every request, so it knows how many are waiting. That is the real demand signal. KEDA reads it from Prometheus (the KEDA Prometheus scaler):

triggers:
  - type: prometheus
    metricType: AverageValue
    metadata:
      query: sum(llm_d_epp_flow_control_queue_size{model_name="..."})
      threshold: "1"   # one waiting request per replica = add a pod

Nothing waiting: you are keeping up. Something waiting: add capacity.

Here is the whole loop:

  1. Requests pile up. KEDA adds a pod.
  2. No free GPU, so the pod sits Pending.
  3. Karpenter sees it, reads what it asked for, and launches a matching GPU node.
  4. Traffic stops. Replicas drop. The node is empty.
  5. About a minute later, Karpenter gives it back (consolidation).

Idle GPU cost is zero. That is the whole point.

The VRAM math

 +------------------ 2 x 96 GB GPU -------------------+
 |  weights ~135 GiB (split)      |  KV cache ~34 GiB  |
 +--------------------------------+--------------------+
        fixed cost                  how many people fit

Weights are a fixed cost. What is left over is KV cache, and that is how many conversations fit at once. Squeeze it and requests queue. That is why queue depth is the right signal - it is a direct measure of running out of KV room.

For Kimi-Dev-72B on two 96 GB GPUs that is about 110,000 tokens of cache. Roughly three full 32k conversations at the same time.

What it looked like

Real numbers from the first run:

Step Time
GPU node launch ~2 min
Download 145 GB of weights ~6 min
Load + CUDA graph capture ~4 min
Cold start, total 12-15 min
Decode speed, one request ~18 tokens/s

The node costs about $8.40 an hour. A two-hour session lands around $17, and a quick proof of concept like mine around $6.

Running it: git is the deploy button

I split the work in two:

  • Terraform + GitHub Actions build the AWS side: VPC, EKS, IAM, Karpenter, KEDA, Prometheus. The pipeline signs in to AWS with OIDC, so no keys are stored anywhere.
  • Argo CD runs everything on the cluster from git: the router, the model server, the node pool, the autoscaler, the chat UI (Open WebUI).

Switching models is a one-line commit: tier: kimi. Setting it to none makes Argo CD prune the lot and Karpenter hand the GPUs back.

One catch worth knowing. With tier: none the app renders nothing, and Argo CD refuses to auto-sync a result that would delete every resource. You need allowEmpty: true on the root app or the shutdown quietly never happens. I only found it by testing the teardown.

The whole thing is open source: llm-d-eks-lab.

Sharp edges

What bit me, so it does not bite you:

  • The default disk is slow. gp3 writes at 125 MB/s. That is about 20 minutes to land 145 GB. Raising it to 1000 MB/s costs around $0.05 an hour and cut the download to 6.
  • Kubernetes gives a rollout 10 minutes. progressDeadlineSeconds defaults to 600. A big model is still loading, perfectly healthy, when the deploy gets marked failed. Derive the deadline from your startup probe.
  • The metric sits behind a feature gate. The EPP only emits queue size with flowControl enabled. Without it the query returns nothing and nothing ever scales - with no error anywhere.
  • The GPU AMI has no device plugin. The EKS NVIDIA image ships drivers, not the NVIDIA device plugin DaemonSet. Without it nodes never advertise nvidia.com/gpu and your pod waits on a node that is already billing.
  • Karpenter reserves karpenter.sh/* tags. Put one in your node class and the whole object is rejected. No node pool, pod Pending forever.
  • vLLM rejects tool_choice: "auto" until you pass --enable-auto-tool-choice and a parser (vLLM tool calling docs). Chat UIs send it by default.
  • llm-d's wins need scale. Prefix-aware routing pays off across many replicas. With one or two you are mostly exercising the control plane. Fine for learning, not a benchmark.

If you are comparing serving layers, I covered the other big option in KServe in plain English.

The whole post in four lines

  • An LLM needs its weights in VRAM, a KV cache per conversation, and a GPU that reads 100% busy either way.
  • llm-d puts a router in front that picks the pod by queue, cache use and prefix hits.
  • KEDA scales pods on queue depth. Karpenter buys GPU nodes for pending pods and returns them when empty.
  • Keep it all in git and it costs about $8 an hour while you use it, nothing after.

The mental model that stuck: llm-d is a load balancer that knows what is in each GPU's memory.

References

llm-d

  • llm-d project site and source on GitHub
  • llm-d request scheduler docs - how the endpoint picker scores pods
  • Intelligent inference scheduling with llm-d
  • Kubernetes Gateway API Inference Extension - the project behind the InferencePool API the router uses

Engine and model

  • vLLM on GitHub and the vLLM docs
  • vLLM: optimization and tuning - tensor parallelism and KV cache sizing
  • vLLM: tool calling - --enable-auto-tool-choice and the parsers
  • Efficient Memory Management for Large Language Model Serving with PagedAttention - the paper behind vLLM's KV cache handling
  • Kimi-Dev-72B model card on Hugging Face, and Open WebUI

Scaling and cluster

  • Kubernetes Deployments (the progress deadline)
  • KEDA and its Prometheus scaler
  • Karpenter, its NodeClasses and disruption (consolidation)
  • Argo CD and its automated sync policy (allowEmpty)
  • Envoy proxy

AWS

  • EBS volume types (gp3)
  • NVIDIA device plugin for Kubernetes
  • GitHub Actions OIDC with AWS

This project

  • llm-d-eks-lab - all the Terraform, Argo CD apps and tiers behind this post
  • KServe in plain English