# Didi Berman — Full Site Content

> Platform Engineer and Kubernetes Security Specialist in Berlin, Germany. CKA, CKAD, CKS, Kubestronaut. I build secure Kubernetes platforms with Terraform, GitOps, and AWS.

Contact: didi@didibe.dev · [GitHub](https://github.com/didiberman) · [LinkedIn](https://www.linkedin.com/in/didibermans)

Certifications: Certified Kubernetes Administrator (CKA); Certified Kubernetes Application Developer (CKAD); Certified Kubernetes Security Specialist (CKS); AWS Solutions Architect Associate (SAA-C03); CNCF Kubestronaut.

This file contains the full text of every blog post on https://didibe.dev. A shorter curated index lives at https://didibe.dev/llms.txt.

## Projects

- [kubewhy](https://github.com/didiberman/kubewhy): AI-powered Kubernetes assistant. An AI-powered, read-only Kubernetes CLI that explains broken workloads in plain English and shows the checks behind each answer.
- [EKS IDP Platform](https://github.com/didiberman/eks-idp-platform): Platform Engineering / AWS EKS. Production IDP on AWS EKS with Terraform modules, ArgoCD, Kyverno, Cilium, Karpenter, and hardened GitHub Actions CI. SHA-pinned actions, step-security harden-runner egress auditing, and Trivy v0.35.0 config scanning on every pipeline run.
- [vcluster-platform](https://github.com/didiberman/vcluster-platform): Multi-Tenant Kubernetes / Platform Engineering. Virtual Kubernetes clusters: the isolation of a real cluster at the cost of a namespace. Each tenant gets its own API server, CRDs, and RBAC while running as a single pod on a shared host. A self-service platform on Hetzner where teams get clusters by opening a pull request: Terraform provisions the host, ArgoCD materializes tenants from merged YAML.
- [Practical AKS](https://github.com/didiberman/practical-aks): DevSecOps / Azure Kubernetes Service. Security built in at every layer: Workload Identity instead of stored credentials, Key Vault for secrets, SHA-pinned CI/CD actions hardened against the March 2026 Trivy supply chain attack, network egress blocking on every pipeline job, containers with zero Linux capabilities. A production-grade AKS cluster from blank Azure subscription to done, with every decision documented.
- [TokenGuard Operator](https://github.com/didiberman/tokenguard-operator): Kubernetes Security. Kubernetes clusters accumulate over-privileged ServiceAccounts over time. TokenGuard is a Go operator that scores each one against its actual audit log usage, and alerts when a token is used from an external IP - usually the first sign of a supply chain compromise.
- [Self-Service Internal Developer Platform](https://github.com/didiberman/kratix-platform): Platform Engineering. A working Internal Developer Platform on k3s, built by wiring together open-source tools: developers request infrastructure through a Backstage portal, Kratix runs pipeline jobs, writes manifests to MinIO, and Flux syncs them to the cluster in seconds. One command provisions the whole stack.
- [GKE Production Patterns](https://github.com/didiberman/gke-production-patterns): Google Kubernetes Engine. Production-grade GKE reference architecture covering the parts that take the longest to get right: Cloud SQL, Memorystore, Helm, CI/CD, observability, Temporal workflows, and incident-response scenarios with intentionally broken environments to debug.
- [NinjaDevOps](https://ninjadevops.com): Live Product. A browser-based platform where you fix real broken servers under time pressure - Linux, Kubernetes, Docker, DNS. Disposable VMs spin up on Compute Engine, connect via CloudFlare Tunnel, and you get instant scoring. Running at ninjadevops.com with Stripe billing.

## Blog Posts

---

# The Argo CD Vulnerability That Spreads From One Pod to Your Entire Cluster

- URL: https://didibe.dev/blog/argocd-repo-server-flaw
- Published: 2026-07-13
- Author: Didi Berman
- Tags: kubernetes, argocd, gitops, security, devops, cloud security

Argo CD has an unpatched security flaw that lets an attacker take over a Kubernetes cluster, and it has been known for 18 months.

Argo CD is one of the most widely used tools for deploying apps onto Kubernetes. Security firm [Synacktiv](https://www.synacktiv.com/en/publications/caught-in-the-octopus-trap-unauthenticated-rce-in-argo-cd-with-codeql) found this flaw and reported it privately to the maintainers in January 2025. There is still no official fix, as of this writing in July 2026. No CVE has even been assigned. If you run Argo CD, this affects you directly, and the fix is something you have to do yourself, today, not something you can wait for a vendor to ship. ([The Hacker News coverage](https://thehackernews.com/2026/07/unpatched-argo-cd-repo-server-flaw.html) is a good plain summary if you want the news version.)

This post explains what Argo CD actually is, what went wrong, and the one thing you can check right now to protect yourself even without a patch.

## First, what is Argo CD?

Most teams that use Kubernetes do not manually type commands to deploy their apps. Instead, they store their app's desired setup as files in a Git repository, and a tool watches that repository and applies changes automatically. This approach is called GitOps.

**Argo CD** is one of the most popular tools for doing this. You push a change to Git, and Argo CD notices, pulls the new files, and updates your Kubernetes cluster to match. It is trusted, widely used, and normally treated as a helpful robot working quietly in the background.

## The piece that broke: repo-server

Argo CD is made of several smaller parts working together. One of them is called **repo-server**. Its job is simple: read the files from your Git repository and turn them into the final instructions Kubernetes understands.

To do this job, repo-server sometimes uses a tool called **kustomize**, which helps combine and customize configuration files. Kustomize, in turn, can call out to another tool called **Helm** to handle a specific kind of configuration package.

Here is the part that matters: kustomize lets you specify exactly which program it should use when it needs Helm, through a setting called `--helm-command`. Normally this points to the real Helm program. Nothing suspicious about that on its own.

## What went wrong

The repo-server component was never checking who was allowed to talk to it internally. There was no login step, no permission check. Anything that could reach it and get it to process a Git repository set up a specific way could hijack what repo-server ran.

One thing to be clear about: this is not an attacker coming in from the internet. In a normal install, repo-server is only reachable from inside the cluster, not exposed to the outside world. The realistic attacker here is something already running inside your cluster, a compromised pod, a leaky application, another workload that got popped, that can quietly reach repo-server because nothing is stopping pods from talking to each other by default. That detail is the whole reason the fix later in this post looks the way it does. Security firm Control Plane makes this exact point in their analysis, aptly titled [Internal ≠ Isolated](https://control-plane.io/posts/argo-cd-repo-server-rce/).

The other thing that matters: this only triggers when an Application is configured to use Kustomize's Helm chart support, where Kustomize renders a Helm chart as part of building the final manifests. Researchers found that an attacker could send repo-server a crafted request that overrode the `--helm-command` setting, the setting that tells Kustomize which program to run as "Helm." Instead of the real Helm program, it pointed to a script from a Git repository the attacker controlled. Synacktiv walks through exactly how they found and built this, including the `GenerateManifest` request they crafted, in their [full technical writeup](https://www.synacktiv.com/en/publications/caught-in-the-octopus-trap-unauthenticated-rce-in-argo-cd-with-codeql).

When repo-server tried to run "Helm," it actually ran the attacker's script instead. And just like that, the attacker's own code was running inside your infrastructure.

## Why that is such a big deal

Once an attacker can run their own code on repo-server, they are not stuck there. Repo-server sits close to the middle of your deployment pipeline, and that comes with access to sensitive things nearby:

- It can read the credentials repo-server holds to do its job, which is a real prize: this is the component that clones your private Git repositories and pulls your private Helm charts, so it has the access tokens and keys for all of them. It also holds the password for Redis, which Argo CD uses as an internal cache.
- It can poison that cache directly. Argo CD stores already-rendered manifests in Redis so it does not have to redo the work of building them from Git every single time.
- The next sync does not always re-read Git and re-render from scratch. If the cache says "here is the manifest to apply," Argo CD can end up applying whatever the attacker planted there instead, without ever touching your real Git repository.

Put plainly: what starts as "I can run one script" can turn into "I now control what runs on your entire Kubernetes cluster."

## The part that should worry you most: there is no patch

Normally a story like this ends with "and then the vendor released a fix, so update now." Not here.

Synacktiv reported this privately in January 2025. As of the article covering this in July 2026, there is still no official patched version. The people who found it even built and held back a tool called `argo-cdown` that automates the full attack, specifically to give organizations more time to protect themselves before releasing it so admins can test their own deployments. This is also why some in the industry now argue GitOps infrastructure like Argo CD should be treated as [tier-zero, meaning as sensitive as your identity provider or your cluster's control plane itself](https://www.csoonline.com/article/4192188/argo-cd-flaw-shows-why-gitops-infrastructure-should-be-treated-as-tier-zero.html).

That is an unusually long gap, and it means "just update Argo CD" is not an option right now. You need a different kind of defense.

## The one setting that actually protects you

The real safety net here is not a patch. It is a basic Kubernetes feature that is often left switched off: **network policies**.

A network policy is a rule that controls which parts of your cluster are allowed to talk to which other parts. Without one, anything inside your cluster can usually reach anything else inside your cluster. With one in place, you can say "only these specific components are allowed to talk to repo-server," and everything else gets blocked.

Network policies are not something specific to Argo CD. They are a standard Kubernetes feature, and it is your cluster's networking layer (called a CNI plugin, examples include Calico and Cilium) that actually enforces them. Argo CD's own official Helm chart has a convenient flag that generates these network policy rules for you, scoped to Argo CD's own components, and that flag ships **turned off by default**. Most installations never turn it on.

One thing to check before you rely on this: not every Kubernetes networking setup enforces network policies at all. Some common ones, like a plain Flannel setup, silently ignore them. Turning the flag on will still create the policy objects, but they will do nothing to protect you unless your cluster's networking layer actually enforces them. Confirm that first, then enable the flag.

That default setting being off is a real part of why this bug is dangerous in practice. Remember, the attacker here is another workload inside your cluster, not someone on the internet. If repo-server and its Redis instance would only accept connections from the other Argo CD components that legitimately need them, a random compromised pod elsewhere in the cluster would have no path to reach the vulnerable part in the first place.

## What to actually do about it

You can check your own cluster right now with one command:

```
kubectl get networkpolicy -A
```

If that comes back mostly empty, or missing anything protecting Argo CD's components, you are likely running exposed.

To fix it:

1. Confirm your cluster's networking layer actually enforces network policies. Calico, Cilium, and most managed Kubernetes CNIs do. Plain Flannel does not, so check yours before assuming you are covered.
2. Turn on the network policy flag in your Argo CD Helm chart configuration. It is usually a simple flag, not a rebuild, and it will generate the right rules scoped to Argo CD's own components.
3. Make sure repo-server and Redis only accept connections from the other Argo CD components that genuinely need to reach them.
4. Check this regularly. A network policy that gets accidentally removed during a future upgrade is a silent way to end up exposed again without noticing.

None of this requires a patch that does not exist yet. It requires flipping a setting that should probably have been on by default in the first place.

## The whole post in three lines

- Argo CD's repo-server component has no internal authentication, so any workload already inside your cluster (not an internet attacker) can send it a crafted request that swaps in a malicious script disguised as the Helm tool and runs their own code inside your pipeline.
- From there, an attacker could read credentials and poison Argo CD's internal manifest cache, getting their own workloads deployed on the next sync without ever touching your real Git repository, and there is still no official patch 18 months after it was reported.
- The real fix available today is enabling Kubernetes network policies (and confirming your cluster's networking layer actually enforces them), which are off by default in Argo CD's Helm chart, so check yours with `kubectl get networkpolicy -A` and lock down repo-server's access.

A tool you trust to deploy your apps is still worth checking on, especially when the fix is a setting you can flip today.

## Sources and further reading

- [Synacktiv: Caught in the Octopus Trap, Unauthenticated RCE in Argo CD with CodeQL](https://www.synacktiv.com/en/publications/caught-in-the-octopus-trap-unauthenticated-rce-in-argo-cd-with-codeql). The original technical writeup from the researchers who found it.
- [The Hacker News: Unpatched Argo CD Repo-Server Flaw Could Let Attackers Take Over Kubernetes Clusters](https://thehackernews.com/2026/07/unpatched-argo-cd-repo-server-flaw.html). The plain-language news summary.
- [Control Plane: Internal is not Isolated (Or Secure)](https://control-plane.io/posts/argo-cd-repo-server-rce/). A deeper security analysis of why in-cluster reachability is the real problem here.
- [CSO Online: Argo CD flaw shows why GitOps infrastructure should be treated as tier zero](https://www.csoonline.com/article/4192188/argo-cd-flaw-shows-why-gitops-infrastructure-should-be-treated-as-tier-zero.html). The broader "treat your deployment tooling as critical infrastructure" argument.

---

# Why Setting Up GPUs for AI Never Quite Works, and What NVIDIA Just Fixed

- URL: https://didibe.dev/blog/nvidia-aicr-vcluster
- Published: 2026-07-10
- Author: Didi Berman
- Tags: kubernetes, nvidia, gpu, vcluster, platform-engineering, ai-infrastructure

This post starts from zero. You do not need to have touched a GPU or Kubernetes before. If you have ever waited for a computer to "install drivers" and had it fail for no visible reason, you already understand the core problem this post is about.

## First: what is a GPU, and why does it need a cluster?

A **GPU** (graphics processing unit) is a chip built to do enormous numbers of small calculations at the same time. Your everyday processor, the CPU, is good at doing one complex thing quickly, one after another. A GPU is good at doing thousands of simple things all at once.

That turns out to be exactly what training and running modern AI models needs. AI models are, underneath everything, giant piles of simple math done over and over. One GPU helps. Many GPUs, working together, help a lot more.

A **cluster** is just a group of machines that act as one. A GPU cluster is a group of servers, each with one or more GPUs inside, wired together so a big AI job can be split across all of them.

## Second: why is Kubernetes involved at all?

**Kubernetes** is the system most companies use to manage groups of servers and the apps running on them. You tell it "I want this running," and it keeps that true, restarting things that crash and placing work on whichever machine has room.

Kubernetes was not originally built with GPUs in mind. A GPU is a physical piece of hardware bolted onto one specific machine, not something Kubernetes understands out of the box the way it understands "give this app some memory." So Kubernetes needs extra help, extra software, to even notice that a GPU exists on a node, let alone hand a slice of it to the right container.

That extra help is where the trouble starts.

## Third: why GPU software is a tower that keeps wobbling

Making a GPU actually usable inside Kubernetes means stacking several pieces of software on top of each other, and every piece has its own version number:

- The **driver**, which lets the operating system talk to the physical GPU chip at all.
- **CUDA**, NVIDIA's toolkit that lets programs actually do math on the GPU.
- **NCCL**, a library that lets multiple GPUs, sometimes on different machines, talk to each other fast, which matters a lot when one AI job is spread across many GPUs.
- The **container runtime**, the software that actually runs your containers.
- The **GPU operator**, extra Kubernetes software that plugs GPUs into the cluster so Kubernetes can hand them out to workloads.
- **Kubernetes itself**, which also has its own version.

Every one of those pieces has to agree with every other piece. Driver version 550 might work fine with CUDA 12.4 but break with CUDA 12.6. A GPU operator update might expect a newer driver than what is actually installed on your machines. None of these pieces update on the same schedule, and none of them ask permission before changing.

## Fourth: the compatibility matrix, and why it lies to you

To help people navigate all this, vendors publish **compatibility matrices**. A big table: driver version down one side, CUDA version across the top, a grid of "yes this combination works" and "no it doesn't."

Here is the problem. That table describes a moment in time, not your cluster right now. A node gets patched. An operator auto-updates. Someone installs a newer kernel to fix an unrelated bug. The table still says what it said yesterday. Your actual cluster has already moved on without telling anyone.

As NVIDIA's own team put it: the matrix is not wrong because people are careless. It is wrong because it is static, and the cluster is not.

Anyone who has ever set up a GPU cluster has hit this exact wall: read the table, follow it exactly, and still watch something fail, because the table was already out of date by the time you read it.

## AICR: stop asking the document, start asking the cluster

NVIDIA's answer is a new open source tool called **AICR**, short for AI Cluster Runtime. The easiest way to think about it: a package manager for your entire GPU stack, in the same spirit as `apt` on Linux or `npm` for JavaScript. Except instead of resolving a list of dependencies from a file someone wrote, it resolves them by actually looking at your cluster, right now, and figuring out from scratch what belongs together.

It runs as a small program plus a background service (`aicrd`), and it works in four steps. None of these require deep GPU knowledge to follow.

**Step 1, Snapshot.** It looks at your cluster and writes down exactly what is really there. Which machines exist, which GPUs they have, which driver versions are installed, which extra software is already running. No guessing, no assumptions, just an honest inventory.

**Step 2, Recipe.** It works out, from that inventory, exactly which versions of everything should be installed together. If you happen to have a specific, well-known setup, like a particular NVIDIA GPU model on a particular cloud provider, it can use a recipe tuned specifically for that combination. If your setup is more generic, it falls back to a sensible default that is still known to work. Either way, you get an answer built for your hardware, not a guess from a table.

**Step 3, Validate.** This is the important part. AICR does not just check that a config file looks correct on paper. It actually runs real test jobs on your cluster, using your real GPUs, to check that everything genuinely works. One of these tests even measures how fast your GPUs can talk to each other, a real performance number, not a guess.

**Step 4, Bundle.** Once everything is confirmed working, it packages the result into whatever format your team already uses to deploy software, so you can roll it out the normal way.

The shift here is simple to explain even if you have never touched a GPU: instead of trusting a document that someone wrote once and hoped stayed true, you trust a test that just ran, on your actual machines, a few seconds ago.

## Where vCluster fits into the story

To demonstrate all of this, NVIDIA did something clever. Instead of using an entire dedicated GPU data center, they used a tool called **vCluster**.

Quick explainer if you have not run into it before: normally, if you want your own private Kubernetes cluster, you need to set up a whole new set of servers just for it, which is slow and expensive. vCluster lets you create what behaves like a fully real, independent Kubernetes cluster, but running inside a lightweight container, on top of infrastructure you already have. Think of it as a virtual machine, but for an entire Kubernetes cluster instead of one computer. I wrote about the idea in more detail in an [earlier post on virtual clusters](/blog/virtual-clusters-explained).

For this demo, NVIDIA used a version called vCluster in Docker, nicknamed **vind**, to spin up a Kubernetes control plane inside a container running on a regular laptop. That tiny control plane then reached out over a secure, encrypted connection and joined a real, physical GPU machine (an NVIDIA A100) sitting far away in Google Cloud. Neither side needed a public internet address for this to work.

So picture it plainly: a laptop, running a lightweight fake-but-fully-functional Kubernetes cluster inside a container, reaching across the internet through a private tunnel, to borrow and command a powerful GPU server it does not own and has never touched before. And it worked.

That is a genuinely useful trick for anyone wanting to test GPU infrastructure without owning a GPU data center themselves. Spin up the control plane locally. Point it at real hardware wherever that hardware happens to live. Tear it down when finished. No leftover cloud bill for a cluster somebody forgot to delete.

## The actual numbers from the demo

For the GPU used in this test, AICR figured out **12 pieces of software that needed to work together, using 3 different sets of rules layered on top of each other** to get there. It then generated **57 files, totaling 135 kilobytes**, ready to deploy.

Then it ran **5 separate real tests** to prove the setup actually worked: whether GPU scheduling worked, whether GPU health monitoring worked, whether AI service monitoring worked, and two more general health checks. All 5 passed. This took about **60 seconds**, and included a real, live reservation of one physical GPU to prove it could actually be used.

Sixty seconds, to prove a brand-new GPU setup genuinely works, instead of hoping an old document is still accurate. That one number is the entire idea of this tool, condensed.

## Why this matters even if you never touch a GPU

You do not need to run AI workloads for the underlying idea here to matter. Any time you rely on a document that describes "what should work," instead of a test that proves "what does work right now," you are trusting something that goes stale the moment reality changes underneath it.

What NVIDIA built here is a small example of a bigger, more useful pattern: look at what is actually there, work out what should go together, prove it with a real test, then package the result. That pattern is worth stealing for other kinds of complicated infrastructure too, not just GPUs.

And the fact that the whole demo could be run from a container on a laptop, reaching across an encrypted tunnel to borrow a GPU sitting somewhere else entirely, says something about how far virtual clusters have come as a genuinely useful piece of infrastructure, not just a way to save money on multi-tenancy.

## The whole post in three lines

- GPU software is a tower of pieces (driver, CUDA, NCCL, Kubernetes, and more) that all have to agree with each other, and that agreement breaks the moment any one piece quietly updates.
- NVIDIA's new tool, AICR, stops trusting old documentation and instead reads your cluster directly, works out what should be installed, and proves it with real tests on real GPUs.
- The demo ran its control plane from a vCluster on a laptop, reached out to a real GPU sitting in the cloud, and proved the whole thing worked in about 60 seconds.

Stop asking an old document if it is still true. Ask the cluster.

---

# An AI Broke Into AWS Faster Than You Can Imagine

- URL: https://didibe.dev/blog/ai-accelerated-aws-compromise
- Published: 2026-07-10
- Author: Didi Berman
- Tags: aws, cloud security, iam, ci/cd, incident response, devsecops

I read something last week that genuinely stopped me for a second. A security firm called Sygnia [investigated an AWS breach](https://www.sygnia.co/blog/inside-an-ai-assisted-cloud-attack/) where the attacker used four different stolen access keys, from four different accounts, all from the same IP address, all within the same observed second.

Read that again. Not the same hour. The same second. I don't know anyone who can type that fast, let alone juggle four separate credentials while doing it.

The full incident went from first foothold to complete takeover in about 72 hours, and that one detail is the reason I wanted to write about it. Most of us build our security assumptions around having some amount of time to react: a key leaks, someone eventually notices, it gets rotated, the blast radius stays small. That assumption is starting to look shaky.

Worth saying up front: there was no zero-day here. The attacker got in the boring way, a leaked secret and a permission that was more generous than it should have been. The same kind of thing sitting in plenty of AWS accounts right now, probably including a few you or I have touched. What's different is what happened after they found it.

This post starts from zero. If you run anything in AWS, you should know what happened and what actually stops it.

## First: what does "compromise an AWS account" mean?

AWS accounts run on **identities** - IAM users, roles, and access keys. An access key is a username and password, but for machines. A script, a pipeline, an app uses it to say "I'm allowed to do this."

AWS can't tell a stolen key from the real owner. The key **is** the permission. Whatever it can do, the attacker can now do.

So the real question isn't "how did they get in." It's "how far could that one key reach."

## How they got the first key

The entry was mundane. They exploited a bug in an internet-facing app and grabbed a credential. Not a cloud misconfiguration on day one - a normal web bug that happened to sit in front of AWS.

From there, they found more credentials in two common places:

- **S3 buckets** holding secrets - config files, `.env` dumps, backups that got stored "temporarily" and never got cleaned up
- **CI/CD environments**, where pipelines routinely hold cloud credentials so they can deploy on your behalf

Neither is exotic. Most teams have something like this somewhere. That's the point.

## The real shift: "waves," not stages

Classic intrusions run in stages. Get in. Look around. Escalate. Move sideways. Do damage. One step at a time.

This one didn't. Sygnia calls it overlapping **waves**. Each new credential immediately triggered its own round of discovery, secret-hunting, and damage - while the last wave was still running.

Think of stage-based hacking as one person searching a building room by room. This was several people entering through different doors at once, each searching their own room, radioing back what they find, while more keep arriving through doors the first group just opened.

The evidence: **four access keys, four different accounts, same IP, same user-agent, same second.** That's not a script running a fixed routine. That's something deciding what to do with each new credential on the spot.

## Why they suspect AI

A few things pointed past a hand-written bot, toward AI-assisted tooling:

- **Adaptation, not repetition.** Several hundred *unique* SQL queries across dozens of databases. A normal tool reuses one query template everywhere. Writing hundreds of distinct, context-appropriate queries at that pace looks like something generating new ones on the fly - the way you'd ask an LLM "here's this schema, write me a query for X."
- **Fast relationship-mapping.** The attacker went from "I have this credential" to "here's everything it touches" almost instantly - work that normally takes a human analyst real time.
- **A strange framing choice.** Some of the attacker's own artifacts described the activity as an authorized "pentest." One theory: if the attacker is driving an AI agent, framing it as an authorized test may be a way to get the agent to comply with actions it would otherwise flag.

None of this is proof. It's forensic inference. But multiple security firms are separately converging on the same read: offense is starting to move at machine speed. Sygnia lays out the AI indicators in detail in [their write-up of the investigation](https://www.sygnia.co/blog/inside-an-ai-assisted-cloud-attack/).

## Not a one-off: the 8-minute Lambda takeover

A [separate incident from November 2025](https://www.sysdig.com/blog/ai-assisted-cloud-intrusion-achieves-admin-access-in-8-minutes), investigated by the Sysdig Threat Research Team, makes the point even sharper. An attacker went from initial access to **full AWS admin control in eight minutes**, via a poisoned Lambda function. The initial credentials were found in public S3 buckets, and AI-driven automation then carried out recon, privilege escalation, and lateral movement across **19 distinct AWS identities** - faster than a coffee break. (The AI fingerprints were almost comical: the tooling tried to assume roles in made-up account IDs like `123456789012`, and left session names like `claude-session` in the logs.)

No zero-day there either. Just stolen credentials, automation, and an environment that let one identity's access snowball into eighteen more.

## What actually made this possible

Strip away the AI framing and the causes are old:

1. **Secrets left in the wrong place.** S3 buckets and CI/CD are the two most common spots for a forgotten credential.
2. **Long-lived credentials.** A key that never expires is valuable forever, to anyone who finds it.
3. **Over-permissioned identities.** If one credential reaches 19 others, one leak becomes a full takeover.
4. **No automated containment.** By the time a human notices and rotates a key by hand, a machine-speed attacker has already finished.

AI didn't invent these weaknesses. It just made chaining all of them, at once, cheap enough to automate. The mistakes are old. The speed is new.

## What actually stops this

**1. Kill long-lived credentials.**

A key that doesn't expire is a permanent liability the moment it's created. Prefer short-lived credentials: IAM roles via STS, OIDC federation from CI/CD (GitHub Actions → AWS, no stored keys at all), instance roles instead of embedded ones. A key that expires in an hour is a much smaller incident than one valid forever.

**2. Scope permissions - and scope who can even assume the role.**

Least privilege is old advice, still correct. But scope both halves: the permissions policy (what this role can do) *and* the trust policy (who's allowed to become this role at all). A role only your `deploy` workflow can assume, touching one bucket and one CloudFront distribution, gives a stolen credential almost nowhere to go.

**3. Get secrets out of S3 and CI/CD env vars.**

Use a real secrets manager with tight IAM access, not config files in a bucket or variables baked into a pipeline. If a secret must live in CI, treat it as radioactive: rotate it often, restrict who reads it, never let it hit logs.

**4. Automate the response, not just detection.**

Sygnia's recommendation worth taking seriously: automate rotation and containment to match attacker speed. If your plan assumes a human notices and rotates a key manually, that window needs to be minutes, not hours - and for machine-speed attacks, minutes might not be enough. CloudTrail alerting on abnormal API velocity, paired with automated key disabling, closes a gap no on-call engineer can close by hand.

**5. Treat identity as the perimeter.**

The old model was "firewall the network." In the cloud, identity *is* the perimeter. MFA everywhere. Aggressive session revocation on anything suspicious. Immediate disabling on any sign of compromise - for machine identities too, which usually reach further than any human ever does.

## The uncomfortable takeaway

Nothing here needed a novel vulnerability. It needed things already sitting in a lot of AWS accounts right now: a secret in a bucket, a credential that never expires, a role that reaches too far. AI didn't invent those problems. It just showed up with the patience and speed to find and chain every one, in a single weekend.

The fix is boring, on purpose: short-lived credentials, tight scoping, secrets out of buckets, detection that doesn't wait on a human. None of it is new advice. What's new is the cost of skipping it, because the thing exploiting the gap doesn't sleep, doesn't get tired, and doesn't work one task at a time.

## The whole post in four lines

- An attacker took over a large AWS environment in ~72 hours using leaked secrets and zero exploits - ordinary entry, extraordinary speed.
- Four access keys used in the same second, plus hundreds of adaptive SQL queries, point to AI-assisted tooling running in parallel "waves," not linear stages.
- A separate incident hit full AWS admin control in **8 minutes** via a compromised Lambda function. Not a one-off.
- **The fix is the same old list, just faster:** kill long-lived credentials, scope permissions and trust policies tightly, get secrets out of buckets and CI, automate detection instead of waiting on a human.

Machine-speed attacks need machine-speed defenses. If your response plan still assumes a human has time to think, it's already out of date.

## Sources and further reading

- [Sygnia: How AI Supercharged a 72-Hour Cloud Attack](https://www.sygnia.co/blog/inside-an-ai-assisted-cloud-attack/). The primary investigation write-up from the firm that responded to the incident.
- [Sygnia press release: AI-accelerated attack lets a lone actor compromise an enterprise cloud](https://www.sygnia.co/press-release/sygnia-investigation-into-ai-accelerated-attack/). The short official summary of the findings.
- [Sysdig: AI-assisted cloud intrusion achieves admin access in 8 minutes](https://www.sysdig.com/blog/ai-assisted-cloud-intrusion-achieves-admin-access-in-8-minutes). The separate November 2025 Lambda incident referenced above.
- [CSO Online: From credentials to cloud admin in 8 minutes](https://www.csoonline.com/article/4126336/from-credentials-to-cloud-admin-in-8-minutes-ai-supercharges-aws-attack-chain.html). Independent reporting on that Lambda incident.
- [Dark Reading: Lone Attacker Uses AI to Breach AWS Cloud Environment in 72 Hours](https://www.darkreading.com/cloud-security/lone-attacker-ai-breach-aws-cloud-environment). Independent reporting on the Sygnia case.

---

# The Trivy CI/CD Hack: Hijacked Pipeline Secrets and the Kubernetes Clusters They Could Reach

- URL: https://didibe.dev/blog/trivy-supply-chain-attack
- Published: 2026-07-05
- Author: Didi Berman
- Tags: Kubernetes security, trivy, supply-chain, github-actions, devsecops, ci/cd

In March 2026, Trivy got hacked. Again.

That sounds like a bad headline. It gets worse when you understand what Trivy is supposed to do.

This post starts from zero. No jargon pile-up. If you run apps on Kubernetes, or you have ever heard the words "CI/CD," you should know what happened and what to do about it.

## First: what is Kubernetes?

Kubernetes (often called K8s) is a system that runs your apps in **containers** across a pool of servers.

You tell it: "I want three copies of my app running." It keeps them running. If one crashes, it starts another.

Most companies do not log into servers by hand anymore. They ship code through automated pipelines. Kubernetes is where that code ends up.

## Next: what is CI/CD?

**CI/CD** means the automated assembly line that builds and ships your software.

Roughly:

1. A developer pushes code to GitHub.
2. A **pipeline** wakes up (often GitHub Actions).
3. The pipeline builds the app, runs tests, scans for security issues, and deploys.

Think of CI/CD as the factory between "code on a laptop" and "app running in production."

That factory holds a lot of keys. Passwords to cloud accounts. Tokens to push container images. Credentials to update the Kubernetes cluster.

## So what is Trivy?

**Trivy** is a free, popular **security scanner**.

It looks at:

- your container images ("does this base image have known bugs?")
- your config files ("is this Kubernetes YAML risky?")
- sometimes your code dependencies

Thousands of teams plug Trivy into CI/CD. It runs automatically on every code change. Before anything reaches Kubernetes, Trivy is supposed to say: "hold on, this has a problem."

It is one of the good guys. A guard at the factory door.

## The paradox: the guard got robbed

That is why this hack stings.

You installed Trivy to **protect** your pipeline. Attackers compromised **Trivy itself** and used that trust to **attack your pipeline**.

The guard became the entry point.

You did not download a shady tool from a forum. You used a well-known scanner from a respected security company (Aqua Security). Your pipeline ran it because best practice said to.

That is the paradox: **the safety tool became the risk.**

## What actually happened (March 2026)

Attackers broke into Trivy's publishing systems. They tampered with three things at once:

1. A bad **Trivy program release** (version 0.69.4)
2. The **GitHub Action** most people use to run Trivy in pipelines (`aquasecurity/trivy-action`)
3. A helper action that **installs** Trivy (`aquasecurity/setup-trivy`)

Here is the sneaky part. They did not ask you to change your config.

They **moved the labels**.

Many pipelines say: "use Trivy version 0.34.2." That version number is a **tag** - a sticky note on a specific piece of code. The attacker peeled off the sticky notes and stuck them on **malicious code instead**. In numbers: they reassigned 76 of the 77 version tags in `trivy-action` and all 7 tags in `setup-trivy` to point at their payload, so nearly every pinned version silently became poisoned at once.

Your file still said `@0.34.2`. The code behind that label was different.

**Analogy:** you saved a contact as "Mom" in your phone. Someone changed the phone number behind "Mom." You tap the same name. You reach a stranger.

The malicious code ran **before** the real scan. It searched the pipeline machine for secrets:

- GitHub tokens
- cloud passwords (AWS, Azure, GCP)
- keys to push container images
- SSH keys
- database passwords in config files

And investigators found it was especially interested in **Kubernetes credentials**.

It hunted for **kubeconfig files** and cluster tokens. More on that in a moment.

Then it encrypted what it found and tried to send it to the attacker over the internet.

This was serious enough that Germany's federal cyber agency (**BSI**) issued an official warning. They rated it **CVSS 9.4 (critical)** and reported real compromise cases linked to the campaign inside Germany. Their advisory explicitly lists **Kubernetes tokens** among the stolen credential types and urges immediate secret rotation if a affected Trivy version ran in your pipelines. Read the full warning here: [BSI Cybersicherheitswarnung 2026-237970-1032 (PDF)](https://www.bsi.bund.de/SharedDocs/Cybersicherheitswarnungen/DE/2026/2026-237970-1032.pdf?__blob=publicationFile&v=4).

For more detail, see [The Hacker News write-up](https://thehackernews.com/2026/03/trivy-security-scanner-github-actions.html) and [Aqua Security's advisory](https://github.com/aquasecurity/trivy/security/advisories/GHSA-69fq-xp46-6x23).

## What is a kubeconfig, and why did attackers want it?

A **kubeconfig** is a small config file. It is the remote control for a Kubernetes cluster.

It usually contains:

- the cluster's address (where to connect)
- a username or service account identity
- a token or certificate (the password proving you are allowed in)

Developers and CI/CD pipelines often keep a kubeconfig on the build machine so the deploy step can say: "apply this update to production."

That file is gold. Whoever holds it can often:

- list everything running in the cluster
- deploy new apps
- read secrets stored inside the cluster
- change permissions

Security researchers analyzing the Trivy malware found it **specifically scanning for kubeconfig files and Kubernetes tokens**. Not as an afterthought. As a target.

Advisories from Aqua Security, [Microsoft](https://www.microsoft.com/en-us/security/blog/2026/03/24/detecting-investigating-defending-against-trivy-supply-chain-compromise/), and others described the stealer sweeping dozens of filesystem paths, dumping CI runner memory, and collecting anything that looked like cluster access - including files under `~/.kube/config` and in-cluster service account tokens.

## Why this matters for Kubernetes clusters

The hack usually started in **CI/CD**, not inside the cluster.

But it did not stop there.

Investigators reported a **methodical follow-on**: stolen kubeconfig files and tokens were used to **log into clusters**, enumerate what was inside, and in some cases deploy persistent backdoors - malicious workloads left running after the pipeline job finished.

[Microsoft's threat research](https://www.microsoft.com/en-us/security/blog/2026/03/24/detecting-investigating-defending-against-trivy-supply-chain-compromise/) noted the same infrastructure was later seen conducting **reconnaissance against Kubernetes environments** (including GKE), issuing Kubernetes API requests with path-traversal payloads to probe for weak cluster endpoints. [Kaspersky](https://www.kaspersky.com/blog/critical-supply-chain-attack-trivy-litellm-checkmarx-teampcp/55510/) tied Trivy, Checkmarx, and litellm together as one campaign and urged victims to check for **suspicious pods** and other signs of cluster compromise, not just leaked secrets in CI.

The attack chain, in plain terms:

1. **Poison Trivy** in your pipeline.
2. **Steal kubeconfig** (and other secrets) from the build machine.
3. **Connect to your cluster** using those stolen credentials.
4. **Explore and infect** - look around, grab more secrets, deploy bad containers if permissions allow.

That is why this is a Kubernetes story, not just a "GitHub got hacked" story.

## How they actually took over a cluster

Researchers now track the group behind this campaign as **TeamPCP**. It turned out Trivy was not their only target, and not their last one.

Three weeks after the Trivy compromise, security firm Endor Labs caught the same group backdooring `litellm`, a hugely popular open-source library (95 million downloads a month) that lets apps talk to different AI providers through one API. Different tool, different package registry (PyPI instead of GitHub Actions), but the Kubernetes attack code inside the payload was nearly identical. That overlap is what let researchers connect the two incidents and describe, in detail, what the malware does once it lands inside a cluster.

Here is the playbook, in plain terms:

1. **Find a Kubernetes service account token on the compromised machine.** CI runners and, in the litellm case, production pods often have one sitting at a well-known file path.
2. **List every node in the cluster.** With that token, the malware calls the Kubernetes API and enumerates every node, workers and control-plane alike.
3. **Deploy a lookalike pod to each one.** Each pod runs in `kube-system` - the namespace where real system components already live, so it blends in. It asks for `privileged: true`, `hostPID`, and `hostNetwork`, and mounts the entire host filesystem into itself. Critically, it **tolerates every taint**, the setting that normally keeps ordinary workloads off control-plane nodes. That one line is what let it schedule anywhere, including nodes that were supposed to be off-limits.
4. **Chroot into the host and drop a backdoor.** With that pod running, the attacker effectively has root on the underlying node, not just inside a container. It installs a small persistence script disguised as a desktop service (a fake "System Telemetry Service" via systemd), then the pod itself terminates and is never restarted, so there is no long-lived workload left in `kubectl get pods` to notice.
5. **Phone home quietly, and know when to go dark.** The backdoor sleeps for five minutes, then checks in with a command-and-control server every 50 minutes for further instructions. Anything it downloads gets a filename designed to look like a routine database process, so it does not stand out in a process list. It even carries a kill switch: if the server ever points it to a page mentioning "youtube," it goes dormant instead of continuing.

None of that needed a Kubernetes vulnerability. It needed one leaked token with permission to create pods, and a cluster that let a pod ask for privileged access, host networking, and every taint toleration, and simply granted it.

That is the uncomfortable lesson. **The privilege escalation was not exploiting a bug in Kubernetes. It was Kubernetes working exactly as configured**, on a cluster that never told a workload "no."

It also reframes the timeline. TeamPCP moved through Aqua Trivy, then Checkmarx's KICS scanner, then npm packages, then `litellm` on PyPI, all inside about a month. Every compromise handed them credentials that opened the next door. The pattern is deliberate: they target security and infrastructure tooling specifically, because that software already carries the elevated access an attacker would otherwise have to fight for. (Full technical writeup: [Endor Labs' analysis of the litellm compromise](https://www.endorlabs.com/learn/teampcp-isnt-done).)

The pipeline often holds the **keys to the kingdom**:

- kubeconfig files with direct cluster admin access
- tokens that can upload a poisoned container image the cluster will happily pull
- access to the Git repo that tells GitOps tools (like Argo CD) what to deploy

Steal a kubeconfig and an attacker may not need to break in through a bug. They use **your own keys**.

Your cluster trusts what CI/CD sends it. If the factory is compromised, production is compromised.

## If you think you were affected: fix it now

If your pipeline ran a compromised Trivy action around **March 19-20, 2026**, assume secrets that job could touch are **stolen**. Do not wait for proof. The BSI warning aligns with vendor guidance: if a compromised version ran, treat **all secrets the pipeline could access as exposed** and rotate them immediately.

**Step 1: rotate secrets - especially kubeconfig and cluster tokens.**

Change passwords and tokens for anything that pipeline could reach. Cloud accounts. GitHub tokens. Registry credentials. Database passwords.

**Rotate every kubeconfig and cluster credential the job could touch.** Revoke old tokens at the cluster level, not just delete the file from CI. Assume the attacker already copied it.

**Step 2: audit the cluster itself.**

Do not only fix CI. Check what happened **inside Kubernetes**:

- unexpected pods or deployments you did not create
- new service accounts or role bindings with broad permissions
- secrets accessed or exported around the same dates
- API audit logs showing logins from unknown IPs

Investigators reported methodical cluster follow-up after kubeconfig theft. Treat the cluster as potentially compromised, not just the pipeline.

**Step 3: check your CI logs.**

Look at pipeline runs from those dates. Weird outbound network traffic is a red flag.

**Step 4: look for a repo called `tpcp-docs`.**

Advisories reported attackers sometimes created a public repo with that name to dump stolen data. If you see it in your GitHub organization, treat it as a confirmed breach.

**Step 5: upgrade to known-safe versions.**

At the time of the advisory, safe references included **trivy-action v0.35.0**, **setup-trivy v0.2.6**, and Trivy binary **v0.69.3**. The BSI and Aqua Security both recommend pinning GitHub Actions to **full commit SHAs**, not movable version tags. Check Aqua Security's advisory for the latest guidance before you pin anything.

Rotating secrets hurts. It is still cheaper than an attacker deploying into your cluster.

## How to prevent the next one

You cannot eliminate risk. You can make attacks much harder.

### 1. Pin tools to exact snapshots, not version labels

Version tags can move. **Commit hashes cannot.**

Instead of trusting `@0.34.2` forever, pin the full commit SHA of the action. Ugly? Yes. Safer? Also yes.

Same idea for the Trivy scanner itself. Pick a version you trust. Do not float on "latest."

### 2. Stop storing kubeconfig in CI

If you can avoid it, do.

Prefer short-lived credentials. Use cloud-native identity (Azure Workload Identity, AWS IAM roles for service accounts) so the pipeline authenticates without a long-lived kubeconfig file sitting on disk.

If CI gets hacked, you want the blast radius small - and you do not want a cluster admin file waiting in `~/.kube/config`.

### 3. Give CI/CD less power

The pipeline should not hold admin keys to everything.

Use short-lived credentials. Give each job only what it needs for that one task. On Azure, **Workload Identity** lets apps authenticate without storing long-lived passwords in GitHub. AWS has similar patterns.

If CI gets hacked, you want the blast radius small.

Here is the catch, though. Short-lived is not the same as safe. A federated credential (OIDC to AWS, Azure, GCP) still exists in memory for the life of the job, and this campaign's malware was dumping runner process memory in real time. It even grabbed AWS credentials straight from the instance metadata service, which are already short-lived by design. If the token is live and the attacker uses it in the same minute it was minted, "expires in 15 minutes" does not save you.

What actually would have stopped the Kubernetes takeover described above is **scope**, not just lifetime. If the identity a Trivy step assumes can only push one image to one registry, a stolen live token gets the attacker a 403 on `kubectl get nodes`, because that permission was never granted in the first place. Two things make this real:

- **Scope the trust policy, not only the permission policy.** With GitHub Actions OIDC, restrict which repo, branch, or workflow file can even assume the role, using the `sub` claim. Not just "what can this role do," but "who is allowed to become this role at all."
- **Split scanning from deploying.** If the Trivy step and the "push to registry" step share one broad role for convenience, compromising the scan compromises the push too. Separate jobs, separate narrowly-scoped identities, and a compromised scan step has nothing downstream to abuse.

Short-lived limits how long a stolen credential is useful. Single-purpose limits what it can do at all while it is still live. You want both.

### 4. Control where CI can call on the internet

Stolen secrets need to phone home. Tools like [StepSecurity harden-runner](https://github.com/step-security/harden-runner) limit which websites a pipeline job can reach. Unknown destinations get blocked or logged.

Malware hates a locked door.

### 5. Keep scanning. Just scan smart.

Do not remove Trivy because it was attacked once. You still need guards at the door.

Just treat the guard like any other piece of software: pin a safe version, verify what you run, and do not assume a famous name means "automatically safe forever."

### 6. Protect Kubernetes even if CI fails

Assume CI might break someday. Inside the cluster:

- block risky configs before they run (policy tools like **Kyverno**)
- specifically, block or restrict **privileged pods, `hostPID`, `hostNetwork`, and broad taint tolerations** for anything outside a short allowlist of real system components. That combination is exactly what let the malware above schedule onto every node, control plane included.
- limit which pods can talk to which (**network policies**)
- require human review before production changes (**GitOps** with pull requests)

Layers. If one wall falls, another still stands. A stolen token is bad. A stolen token that still cannot deploy a privileged pod to your control plane is a much smaller incident.

## What I am doing about it

I updated two public platform projects on my GitHub to bake these lessons in.

**[Practical AKS](https://github.com/didiberman/practical-aks)** - an Azure Kubernetes cluster built step by step. The CI pipeline uses SHA-pinned GitHub Actions (so tags cannot silently move), harden-runner to control outbound traffic, Workload Identity instead of stored cloud passwords, Azure Key Vault for secrets, strict container permissions, and Trivy scanning on pinned safe versions.

**[EKS IDP Platform](https://github.com/didiberman/eks-idp-platform)** - a similar story on AWS EKS. Same CI hardening, plus cluster guardrails: Kyverno policies, Cilium networking, GitOps with Argo CD so deploys stay reviewable.

The point is not perfection. The point is **the path from laptop to cluster should not be the softest target in the building.**

Your scanner sits on that path. It sees the same secrets your deploy step uses. After March 2026, that is obvious. Treat CI/CD like part of production security, not a side alley.

## The whole post in four lines

- **Trivy** scans software for vulnerabilities before it ships. In March 2026, attackers hacked Trivy, stole CI/CD secrets, and **hunted kubeconfig files to break into Kubernetes clusters**.
- The malware did not stop at the pipeline. Stolen credentials were used to **deploy privileged pods to every node**, tolerating every taint to reach even the control plane, then drop a quiet backdoor.
- The same group (**TeamPCP**) reused the identical Kubernetes attack code weeks later in an unrelated `litellm` supply chain compromise, this is a repeatable playbook, not a one-off.
- **Fix it:** rotate kubeconfig and cluster tokens, audit the cluster for suspicious pods, check CI logs, upgrade to safe versions.
- **Prevent it:** pin exact tool versions, keep kubeconfig out of CI, limit permissions, control outbound traffic, and keep cluster policies as a backup wall.

The guard at the door needs guarding too.

## Sources and further reading

- [Aqua Security advisory: Trivy ecosystem supply chain temporarily compromised](https://github.com/aquasecurity/trivy/security/advisories/GHSA-69fq-xp46-6x23). The vendor's own advisory and remediation guidance.
- [Microsoft Security: Detecting, investigating, and defending against the Trivy supply chain compromise](https://www.microsoft.com/en-us/security/blog/2026/03/24/detecting-investigating-defending-against-trivy-supply-chain-compromise/). Detection guidance plus the GKE reconnaissance findings.
- [Wiz: Trivy Compromised by TeamPCP](https://www.wiz.io/blog/trivy-compromised-teampcp-supply-chain-attack). Attribution and timeline of the TeamPCP campaign.
- [Socket: Trivy Under Attack Again, GitHub Actions Tag Compromise](https://socket.dev/blog/trivy-under-attack-again-github-actions-compromise). The tag-reassignment mechanics (76 of 77 tags) in detail.
- [Kaspersky: Trojanization of Trivy, Checkmarx, and LiteLLM](https://www.kaspersky.com/blog/critical-supply-chain-attack-trivy-litellm-checkmarx-teampcp/55510/). Ties the separate compromises into one campaign.
- [Endor Labs: TeamPCP isn't done](https://www.endorlabs.com/learn/teampcp-isnt-done). The technical analysis of the Kubernetes takeover payload via the litellm compromise.
- [BSI Cybersicherheitswarnung 2026-237970-1032 (PDF)](https://www.bsi.bund.de/SharedDocs/Cybersicherheitswarnungen/DE/2026/2026-237970-1032.pdf?__blob=publicationFile&v=4). Germany's federal cyber agency advisory (CVSS 9.4).

---

# KServe in Plain English: Model Serving on Kubernetes

- URL: https://didibe.dev/blog/kserve-inference-made-simple
- Published: 2026-07-04
- Author: Didi Berman
- Tags: mlops, kserve, kubernetes, inference, machine-learning, knative

I'm dipping my fingers into KServe right now. Not because I love another YAML file. Because model serving on Kubernetes has its own shape, and I wanted to understand it before wiring anything up.

## The problem KServe solves

Somewhere upstream, a team produces a model - files on disk, weights, config. Production wants an HTTP API around it that:

- stays up when a node dies
- scales when traffic spikes
- lets you ship version 2 without a scary big-bang cutover
- maybe scales to zero when nobody is calling it

Without a serving framework, teams usually stitch that together from Deployments, Services, Ingress, HPA, and custom glue. It works. It also gets repetitive - the same patterns, rebuilt for every new model.

KServe is the shortcut. It is a Kubernetes-native framework for model serving. You describe what you want. Controllers do the boring parts.

Same pattern as everything else in Kubernetes: objects, robots, reconciliation loop. If that idea is new, start with [my Kubernetes for dummies post](/blog/kubernetes-for-dummies).

## What KServe actually is

Think of KServe as a **restaurant kitchen for models**.

- The **model** is the recipe and ingredients (weights, config, runtime).
- **KServe** is the kitchen staff: prep station, pass window, health checks, surge staffing when the lunch rush hits.
- **Users** order at the counter (send JSON, get a prediction back).

You do not tell the kitchen how many burners to light. You put in an order slip. The system figures out the rest.

Under the hood, KServe is a set of **Custom Resources** (new object types) plus controllers that watch them. The main one to learn first is an **InferenceService**.

## The InferenceService: one object, one model API

An InferenceService is your wish, written down:

> "Run this model. Expose it at this URL. Keep it healthy. Scale it sensibly."

You apply one YAML. KServe creates the pods, the networking, and the routing glue.

A typical InferenceService has up to three logical parts:

1. **Predictor** - the model server itself. The thing that runs inference. TensorFlow, PyTorch, scikit-learn, ONNX, Triton, pick your runtime.
2. **Transformer** (optional) - pre/post-processing. Tokenize text before the model sees it. Format the response after.
3. **Explainer** (optional) - "why did the model say that?" Useful for debugging and compliance.

Most getting-started paths use just a **predictor**. One box. One job. Start there.

## How a request flows

Short version:

1. User sends HTTP POST with input data.
2. Traffic hits the InferenceService endpoint (Ingress or Istio/Knative gateway, depending on your setup).
3. KServe routes to a healthy predictor pod.
4. Pod runs the model. Returns JSON.
5. If traffic doubles, KServe scales predictor pods up. If traffic drops to zero (and you enabled scale-to-zero), pods can sleep.

That last part is why KServe often sits on top of **Knative Serving**. Knative knows how to scale based on **request concurrency**, not just CPU. For inference workloads, that matters. GPUs sitting idle still cost money.

## Why not just a Deployment?

Fair question. A Deployment says "run N copies of this container." It does not know:

- this container is a **model** with a standard predict URL
- version A should get 10% of traffic and version B should get 90%
- scale when **requests queue up**, not when CPU hits 70%
- wire in a transformer stage before the predictor

KServe encodes those patterns as **defaults**. You get canary rollouts, autoscaling, and a consistent serving contract without building a mini platform in your repo.

## What it looks like in practice (conceptually)

You install KServe on a cluster (Helm chart, operator, docs walk you through it). Then something like:

```yaml
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: hello-model
spec:
  predictor:
    sklearn:
      storageUri: "gs://my-bucket/models/v1"
```

`storageUri` points at model artifacts - S3, GCS, PVC, whatever the cluster can reach. KServe pulls them, starts the right runtime, exposes an endpoint.

The exact YAML varies by framework. The **shape** is always the same: one InferenceService, one predictor block, one storage location.

## Sharp edges

A few things worth knowing before you treat it as done:

- **Cluster dependencies.** KServe wants a serving layer (often Knative) and sometimes a service mesh. Read the install docs for your environment before you assume `kubectl apply` is the whole story.
- **Model format vs runtime.** A PyTorch export and a sklearn pickle need different predictor types. Match the block to what was actually packaged.
- **Cold starts.** Scale-to-zero saves money. First request after idle can be slow while a pod wakes up. Fine for demos. Worth planning for in production.
- **Observability.** You still need logs, metrics, and traces around InferenceServices. KServe does not replace a monitoring stack.

None of that means KServe is wrong. It means it is **infrastructure**, not magic.

## When KServe is worth it

Reach for KServe when:

- you are serving models **on Kubernetes** (not Lambda, not a managed SageMaker endpoint)
- more than one model is coming, and you do not want copy-paste Deployments for each
- you care about **version rollouts** and **request-aware scaling**

Skip it (for now) when:

- you have one model, low traffic, and a simple Deployment already covers it
- you are not on Kubernetes
- you need a full ML training pipeline - KServe is **serving**, not training

## The whole post in four lines

- Models arrive as files. Production wants a reliable HTTP API around them.
- KServe is Kubernetes-native model serving: describe an InferenceService, get endpoints, health checks, scaling, and rollouts.
- The predictor runs the model. Transformer and explainer are optional extras.
- It exists so teams do not rebuild the same serving plumbing for every new model.

The mental model clicked fast: **KServe is Deployments for ML inference, with the ML opinions baked in.**

---

# Virtual Clusters, Explained

- URL: https://didibe.dev/blog/virtual-clusters-explained
- Published: 2026-07-03
- Author: Didi Berman
- Tags: kubernetes, vcluster, multi-tenancy, platform-engineering

Every Kubernetes platform team eventually hits the same wall. Multiple teams want to share your infrastructure. What does each team actually get?

## The two classic answers, and why both hurt

**Option one: a dedicated cluster per team.** The waking world. Full physical isolation and independent machines, but building a brand new world for every team from scratch is slow, heavy, and extremely expensive.

The costs pile up fast. Every managed cluster bills a control plane fee before a single app runs. Every cluster needs its own nodes, its own upgrades, its own monitoring. Provisioning one takes around twenty minutes. Ten teams means ten clusters to patch on security-release day.

**Option two: a namespace per team.** Sharing a single dream space. Cheap and efficient, but the walls are paper-thin. When you share a dream level, one team's subconscious projections can easily bleed into another's.

Cheap, but the mental walls are leaky. And here's the part people underestimate: the leaks aren't where you expect.

- **CRDs are cluster-wide.** Custom resource types can't be namespaced. If Team A installs version 2 of an operator and Team B needs version 1, someone loses.
- **One API server for everyone.** A misbehaving controller from one team can hammer the API server that all teams depend on.
- **One Kubernetes version for everyone.** Nobody can test the next release without a separate cluster anyway.
- **One blast radius.** A bad admission webhook installed by one team can block deployments for every team.

So the platform team gets to choose: pay for massive physical worlds nobody fully uses, or play referee when roommate projections collide in a shared dream. For years these were the only choices.

## The middle path: a cluster inside a cluster

A virtual cluster (I use the [vcluster](https://www.vcluster.com/) project) is a complete Kubernetes control plane that runs as a single pod inside a host cluster.

Read that again, because it sounds stranger than it is. The thing your team connects to is a real API server, with its own database, its own CRDs, its own RBAC, its own Kubernetes version. It passes conformance tests. Your tooling can't tell the difference.

But from the host cluster's point of view, that entire "cluster" is just a namespace containing a couple of pods.

Think of it like a **dream within a dream**. The tenant gets their own self-contained dream level with its own physics (CRDs), users, and rules. To the tenant's tools, the dream is entirely real. But to the host cluster - the waking world - the entire dream level is just a single sleeper (a pod) resting in a corner of a namespace.

## How it works: the syncer

A control plane alone can't run anything. It's a database of wishes with nobody to fulfil them (the reconciliation loop idea again). So where do the actual workloads go?

A component called the **syncer** acts as the architect bridging the dream world and physical reality. When you design a pod inside the dream (virtual cluster), the syncer projects it down into the physical world (the host cluster) to run on real hardware. It then mirrors the status back up, so to the user inside the dream, the pod appears to be running on their own dedicated servers.

Everything else stays private. Deployments, CRDs, RBAC roles, secrets: they live only in the virtual cluster's private subconscious (its own database). The host never sees them. Only the things that must physically exist (pods, their services) get projected down.

That split is the whole trick. The expensive, conflict-prone parts of a cluster (API server, CRDs, versions) are duplicated per tenant, because they're cheap to duplicate. The genuinely expensive part (compute) stays shared.

## What virtual clusters don't solve

Honesty section. The isolation is real at the API layer, not the hardware layer.

- **Workloads still share nodes and a kernel.** Against merely clumsy neighbours, that's fine. Against actively hostile tenants, you still need the heavier tools: isolated node pools, sandboxed runtimes, real network policy.
- **Compute contention doesn't vanish.** A tenant that eats all the CPU still hurts everyone. Resource quotas remain your friend.
- **The host cluster is still a single point of failure.** If it goes down, every virtual cluster inside it goes down too. You've consolidated blast radius for control planes, not eliminated it.
- **Networking is shared.** Cross-tenant traffic control still needs deliberate design.

Virtual clusters remove the *organisational* conflicts of shared tenancy: CRD versions, API server access, upgrade schedules, admission webhooks. The *physical* concerns stay exactly where they were.

---

# Four Ways Organisations Build Internal Developer Platforms

- URL: https://didibe.dev/blog/idp-approaches-compared
- Published: 2026-07-03
- Author: Didi Berman
- Tags: platform-engineering, idp, kubernetes, backstage, crossplane, gitops

Here's the problem in one line. Engineers constantly need things: a database, a test environment, a way to ship code. Getting them usually means asking another team. Then waiting.

That waiting is one of the most expensive hidden costs in tech. Highly paid people, blocked on internal bureaucracy.

An Internal Developer Platform (IDP) is whatever a company builds to end the waiting.

But ask five companies to show you "the platform" and you'll see five different things. A wiki full of snippets. An ancient build server. A shiny portal. A Slack channel called #infra-requests.

These aren't five maturity levels of one thing. They're four different strategies. I've built working versions of each (they're on my GitHub). Here's the honest comparison.

One analogy carries us through: how do you feed your employees?

- **Ticket queue** = a canteen clerk takes your order. Come back Thursday.
- **Golden modules** = a warehouse of meal kits. Great ingredients, you cook.
- **GitOps** = an order form that a robot kitchen fulfils. Exactly. Slowly.
- **Portal** = a vending machine. Press button, get meal.

## Level 0: The ticket queue

No platform. Just an ops team and a queue. Need a database? File a ticket. Someone with production access makes it exist.

**Pluses:** Zero engineering investment. A human reviews every change. For a five-person company, this is correct.

**Minuses:** It scales with headcount, not tooling. Every request interrupts one person and blocks another. And the knowledge lives in heads, not code. Every departure is an outage risk.

Everyone agrees this stage should end. The fight is over what replaces it.

## Approach 1: Platform as a library (golden modules)

The first real move for most orgs. The platform team ships high-quality building blocks. Developers assemble them.

The blocks are things like Terraform modules (Terraform: code that describes servers, so infrastructure can be reviewed like software) and ready-made deploy pipelines. This is the meal-kit warehouse. Great ingredients. You still cook.

My [gke-production-patterns](https://github.com/didiberman/gke-production-patterns) and [practical-aks](https://github.com/didiberman/practical-aks) repos are built this way. Opinionated, production-shaped, meant to be copied.

**Pluses:**

- Low buy-in. Teams adopt one block at a time.
- Everything is versioned and reviewable, like any code.
- The platform team stays small. They ship artifacts, not a service.

**Minuses:**

- Developers still have to cook. Using the library means understanding Terraform, cloud permissions, and plumbing. The friction moved. It didn't disappear.
- Copy-paste drift. Team B forks Team A's module and quietly edits it. Eighteen months later, "the standard" describes nothing in production.
- No live inventory. The library says how things *should* be built. Nobody can answer: what's actually running, and who owns it?

## Approach 2: Platform as a pull request (GitOps)

Now the interface changes. Developers stop running tools. Instead, they edit a shared repository that describes what should exist. A teammate approves the change (a "pull request"). Then software like ArgoCD or Flux reshapes the real infrastructure to match. Automatically.

This is the robot kitchen. Fill in the form. Get a sign-off. The machine cooks exactly what the form says.

My [terraform-k8s-argocd-cicd-pipeline](https://github.com/didiberman/terraform-k8s-argocd-cicd-pipeline) repo runs this way. [vcluster-platform](https://github.com/didiberman/vcluster-platform) goes further: every team gets a private Kubernetes environment, provisioned by pull request.

**Pluses:**

- The audit trail is the change history itself. Every production change has a name, a time, and a reviewer. Compliance conversations get short.
- Undo is built in. Revert the change, the robot un-cooks the meal.
- No UI to build. The platform team ships automation and repo structure. Standard dev tooling does the rest.

**Minuses:**

- Slow, indirect feedback. You submit the form. You wait. If something breaks, you go hunting through the robot's status screens. Nothing you did appears to *do* anything. Newcomers hate this.
- The paperwork gets thick. A hundred services, four environments each. That's a mountain of config files nobody can navigate without tribal knowledge.
- Poor discoverability. The repo shows what's declared, not what's on the menu. "How do I get a database?" still gets asked in Slack.

## Approach 3: Platform as a product (portal + control plane)

The full IDP. A self-service portal in front. An orchestration engine behind it. This is the vending machine: press button, get meal.

I built an end-to-end version in [kratix-platform](https://github.com/didiberman/kratix-platform). Backstage (an open-source portal from Spotify) is the storefront. Kratix is the engine that turns a button press into running infrastructure. A developer clicks a template. Ninety seconds later a database exists, catalogued, visible on their dashboard.

**Pluses:**

- Actual self-service. The golden path is a form, not a runbook. Nobody needs to understand the machinery.
- The catalog finally answers "what exists and who owns it."
- You can run the platform like a product. Measure adoption. Retire old offerings. Evolve interfaces like contracts.

**Minuses:**

- You now operate a complex system whose only customer is your own company. My hardest bug in the Kratix build: one component checked its storage backend a few seconds too early, gave up, and never retried. Every status light stayed green. Nothing ever came out. The vending machine broke, silently, while insisting it was fine. That's the class of problem you own at this level.
- Backstage is a full-time job. It's a framework, not a finished app. Expect custom plugins and upgrades forever.
- The floor is high. Approaches 1 and 2 deliver value with one engineer. This needs a team. And if the vending machine is even slightly less reliable than the clerk it replaced, developers will walk past it. Then you maintain both.

## The housing question underneath all of this

Whatever the interface, a Kubernetes platform must decide: what does each team actually *get*?

The classic options are extremes. A dedicated cluster per team is a private house. Full privacy, painful cost, slow to build. A shared namespace is a room in a big shared house. Nearly free, but everyone shares the plumbing. And the blast radius.

The middle path, which I explore in vcluster-platform, is the virtual cluster. Technically: a full private Kubernetes control plane running as a small program inside a shared host. The team sees their own building, their own keys, their own rules. The landlord sees one modest tenant using a third of a gigabyte of memory.

A very good sublet, at one percent of the rent. It makes "every team gets a cluster" affordable for orgs that could never justify it before.

## So which approach is right?

They're not competing philosophies. They're stages of earned complexity.

1. Start with meal kits. Ship building blocks the moment the ticket queue hurts.
2. Move to the robot kitchen when you need audit trails and a single source of truth more than speed.
3. Build the vending machine only when developers are drowning in the first two, and you have a team ready to run the platform as a real product.

The failure mode: skipping the evidence. Building a portal because conference talks made it look mandatory, when the actual bottleneck was three missing Terraform modules.

The platform is not the portal, the pipeline, or the cluster. It's whatever removes the ticket queue without becoming a worse one.

---

# Kubernetes for Dummies: It's Just Objects All the Way Down

- URL: https://didibe.dev/blog/kubernetes-for-dummies
- Published: 2026-07-02
- Author: Didi Berman
- Tags: kubernetes, beginners, reconciliation, learning

Kubernetes has a reputation: too big, too complex, too many YAML files. I remember staring at my first cluster thinking I'd never hold it all in my head.

Then the model clicked. And the model is embarrassingly simple.

Kubernetes is two things:

1. A database full of objects.
2. Robots that make reality match the database.

That's it. Everything else is detail.

## You never command Kubernetes

This is the part nobody tells beginners, and it's the part that matters most.

You never tell Kubernetes to *do* something. You tell it what you *want*, by writing an object into its database. Then you walk away.

An object is just a record. A wish, written down. "I want three copies of my app running." You submit that wish. You don't start the app. You don't pick a server. You describe the end state and stop.

Compare that to every tool you've used before. A shell script is a list of commands: do this, then this, then this. If step four fails at 3am, the script is done and so is your app. Kubernetes doesn't work like that. There are no steps. There's only the wish, sitting in the database, forever.

## The robots

So who does the actual work? Controllers. I think of them as small, obsessive robots.

Each robot has one job and runs one loop, forever:

1. Read the wish. ("Three copies should exist.")
2. Look at reality. ("Two are running.")
3. Fix the difference. (Start one more.)
4. Go back to step 1.

This loop is called the **reconciliation loop**, and it is the single most important idea in Kubernetes. Not pods. Not YAML. This loop.

Your home thermostat is a reconciliation loop. You set 21 degrees. It measures the room. Too cold, heat on. Warm enough, heat off. It never finishes. It never celebrates. It just keeps comparing wish to reality and nudging.

Now the 3am story again. A server dies and takes your app with it. No pager goes off. No script re-runs. A robot simply notices, on its next loop, that reality says two and the wish says three. So it starts a third. That's Kubernetes' famous "self-healing," and it's not magic. It's a thermostat.

Delete a pod by hand and watch it come back seconds later. People find this spooky at first. It's just the loop: you changed reality, but you didn't change the wish.

## Never-ending objects, each with one tiny job

Here's what overwhelmed me early on: the sheer number of object types. Pod, Deployment, ReplicaSet, Service, Ingress, ConfigMap, Secret, Namespace, and on and on. It feels infinite. It basically is.

The relief comes when you notice each object has one small, boring purpose:

- A **Pod** is a running copy of your app. The unit of "actually alive."
- A **Deployment** is the wish "keep N copies of this pod running, and replace them gently when I ship a new version."
- A **Service** is a stable phone number. Pods die and get new addresses constantly; the Service is the one number that always reaches whoever's alive.
- An **Ingress** is the front door: it routes traffic from the outside world to a Service.
- A **ConfigMap** holds settings. A **Secret** holds passwords. Pods reference them instead of hard-coding values.

None of these is complicated alone. A Deployment doesn't route traffic. A Service doesn't run anything. Small objects, small jobs.

## How they relate: loose references, not ownership

The objects form chains, but they connect in the loosest possible way: mostly by labels, which are just sticky notes.

A Service doesn't contain a list of pods. It says "I point at anything labeled `app: shop`." Pods carry the label. That's the whole relationship. New pod appears with the right sticky note, the Service starts routing to it. Pod dies, it drops out. Nobody updates a registry, because there is no registry. Just labels and loops.

The chains look like this:

- You write a **Deployment**. Its robot creates a **ReplicaSet** (the wish "exactly N of this exact pod"). The ReplicaSet's robot creates the **Pods**.
- A **Service** finds those pods by label and gives them one stable address.
- An **Ingress** points at the Service and opens it to the world.

Notice something: even the objects create objects. You wish for a Deployment, and robots write two more layers of wishes on your behalf. Objects all the way down.

## How I actually learned it

I stopped memorizing kubectl commands. Commands are surface. Instead, every time I met a new object type, I asked the same three questions:

1. **What wish does this object express?** One sentence. If I can't say it in one sentence, I don't understand it yet.
2. **What does it point to, and what points to it?** Labels, names, references. Draw the arrows.
3. **Which robot watches it, and what does that robot do when reality drifts?**

Three questions, every object, no exceptions. Ingress? Wish: route this hostname to that Service. Points to: a Service by name. Robot: the ingress controller, which reconfigures the actual load balancer. Done. Next object.

This works because Kubernetes has exactly one design pattern, reused everywhere. Learn the pattern once and every new object is fifteen minutes instead of a weekend.

## Why "never-ending" is the point

The object list isn't just long. It's deliberately open. Kubernetes lets anyone define new object types (called Custom Resource Definitions, or CRDs) and write new robots to watch them.

That means the pattern extends beyond Kubernetes itself. Want a "PostgresDatabase" object, so developers can wish for a database the same way they wish for three pods? You can build that: define the object, write the robot. This is exactly how the platform tooling in [my previous post](/blog/idp-approaches-compared) works under the hood. Crossplane, Kratix, all of it: new objects, new robots, same loop.

Which is why the mental model matters more than any command. Kubernetes isn't a container tool that got out of hand. It's a general machine for turning written-down wishes into reality, and containers were just the first wish.

## The whole post in four lines

- Kubernetes is a database of objects plus robots that make reality match it.
- You never command it. You write wishes and walk away.
- Every robot runs the same never-ending loop: read the wish, check reality, fix the gap.
- Every object has one tiny job, and they connect with sticky notes.

Hold onto that, and the YAML is just paperwork.