Since v1.35 the kubelet setting failCgroupV1 defaults to true, so on a node still running cgroup v1 the kubelet does not start. kubeadm's SystemVerification preflight also errors on init/join/upgrade once the kubelet is 1.35+. On a control-plane node that means the static pods (API server, etcd) do not come back after a reboot either.

Check a node:

stat -fc %T /sys/fs/cgroup/

cgroup2fs = v2, fine. tmpfs = v1.

Check every node from your laptop, no SSH:

for n in $(kubectl get nodes -o name); do
  printf "%s " "$n"
  kubectl debug "$n" -n default -q -i --profile=general --image=busybox \
    -- chroot /host stat -fc %T /sys/fs/cgroup
done

Two gotchas from testing it on a 1.35 kind cluster:

• Keep the chroot /host. Busybox's own stat does not know the cgroup2 magic and prints UNKNOWN on a healthy v2 node.
• It leaves one node-debugger-* pod per node behind: kubectl get pods -n default -o name | grep node-debugger | xargs kubectl delete -n default

Defaults to v1: CentOS 7, RHEL 7/8, Ubuntu 18.04/20.04, Amazon Linux 2. Most of those ship a kernel below the documented 5.8 minimum, and AWS does not support v2 on AL2 at all, so a kernel argument will not fix them. They need a new OS image.

failCgroupV1: false is the escape hatch; KEP-5573 slates removal for no earlier than 1.38.

I wrote up the full checklist (requirements, driver checks, in-place switch vs replace) here: https://saipisey.com/blog/cgroup-v1-kubelet-check

Happy to be corrected if your distro behaves differently.

They also note that overcoming these limitations is feasible, but requires "serious work", including rewriting or replacing etcd and parallelizing the scheduling algorithm.

In this case, can you see a world where there'll be a version of etcd or other components for reinforcement learning tasks and/or anything involving sandbox creation at scale? Or do you think a load of companies, like Modal, are just going to rewrite each of these components until they work for their use case? Genuinely curious.

Happy Autumn k8s folks! (Unless you're down under then Happy Spring~)

Join us on Thursday, 10/29 for another double-header meetup and you'll get to hear from Lesley Cordero and Michael Guarino.

👀 Lesley will be leading a session on "Who Watches the Watchers? Data Governance for Observability"

Lesley's Session Abstract: As organizations invest more in observability, application telemetry (metrics, logs, and traces) has become voluminous and operationally critical. As applications decompose into microservices, silos and telemetry gaps proliferate. We’ll explore data governance for application telemetry: shared specifications, standard context propagation, and standardized platform components for instrumentation, collection, and analysis. We’ll discuss real-world risks and tradeoffs around privacy, cost, access control, and data lifecycle management, alongside strategies for balancing developer productivity with governance.

✍️ Michael will share lessons from rewriting Prometheus / Loki / Tempo in Rust.

Michael's Session Abstract: We recently did a major spike on rewriting the grafana LGTM stack from go to rust, on top of a new object store native database slatedb. We'll go into some lessons learned, which spans diving into the internals of the new breed of object store backed datastores and pushing agentic development to its limit.

RSVP at https://luma.com/05srus23

Let me know if you have any questions - this is a free event and we welcome new folks :)

Currently at work we have an argocd cluster with all our clusters. We have our clusters split with environments.

When we need to test our changes before opening a pr to merge we have to create a whole new cluster that runs argo on it. Making sure it deploys as it should. We do this so it's a 1:1 with how our live clusters are like. However, it takes 45 min to spin up a new cluster. No problem, just have one cluster we all use but then we run into a problem that if multiple people are testing changes to their addon, we are not testing only our changes.

All the blog posts online just run a local cluster with Argo to test. This wouldn't mimic our vpc, iam roles, etc.

So how do all of you test your changes with argocd?

i deploy jupyterhubs to california institutions for free under a grant from the state's startup incubator... so i figured a monitor w/a jupyter notebook, some granfana and particularly some k9s love was in order for our big-ass monitor! :)

https://imgur.com/a/UvyfTim

(i use old reddit, so i can't really embed images... unless there's a PEBKAC lol)

edit: removed accidental duplicate text pasting

Omni’s hobby plan is free now: up to 10 nodes and 2 users (it used to be $10 a month for 6 nodes). Nodes dial out over WireGuard, so nothing is open on the router, and you get upgrades from the UI, SSO for kubectl and talosctl, and scheduled etcd backups.

You can import a running Talos cluster too. I moved my 3-node one without resetting a node. One catch: after the CA rotation, admin access only works through Omni.

How I did the import: https://blog.prateekjain.dev/importing-an-existing-talos-cluster-into-omni-bb1fc7722eda?sk=a4c4412829ef81542ca4892a0edbecec

Commands and patches: https://github.com/prateekjaindev/homelab

Not affiliated with Sidero Labs, just a user. The write-up and repo are my own.

I am designing a service that can spin up a requested database snapshot (Neo4j) and pass queries to the database via the requester's cypher payload -> bolt. These database instances are ephemeral, and they should die after some amount of inactivity.

I've never done any sort of container orchestration, but my initial thought was to have a small HTTP server that controls the ephemeral database instances and incoming cypher requests. This and all ephemeral database containers (1-3 concurrent instances would be typical, maximum of 5), would live on a single VM or app service.

I work at a startup and we need a lightweight solution to this problem that could scale within the next 6 months to year. What I'm proposing now is cheap and simple. I think i can get what i need with k3s (1 control plane and up to 10 nodes) running on the same VM as the HTTP server. But Codex is pushing AKS or ACI. I really don't think that's necessary with the amount of business we are doing right now. And I think that if i design the interface properly, I can easily move from k3s -> AKS later without having to redesign the server or change its API schema.

What do you all think? Again, very new to k8s. Maybe i need to read a book first.

The built-in view role skips Secrets but still reads ConfigMaps and pod logs, and both can end up holding credentials. For a bot that helps with on-call debugging, what did you grant it, and did you scope it per namespace?

Did you learn something new this week? Share here!

Been running home labs for a long time. This is the first time I rebuilt everything on kubernetes and it has been great!

One of the fun things about the fact that all of us wrangling service meshes end up knowing each other is that it opens doors to sitting down with one’s competitors for actual conversations – and tomorrow at noon US/Eastern, my colleagues Keith Mattix, Sarah Christoff, and I will be doing exactly that. Keith’s an Istio maintainer; Sarah and I work with Linkerd, and this’ll be part fireside chat, part crystal-ball gazing, and part AMA.

Bring your questions! and expect real - but pleasant - disagreements about a whole lot of things. 🙂

https://www.buoyant.io/register/the-runbook-crystal-ball-october-2026

As the title suggests, im interested in learning ai infra. Serving a model, kv cache, vllm etc. Im not finding any tutorials that covers all tjis stuff in order.
Can you guys help me where to start?

say a workload uses a quarter of what it requests. cutting the request seems obvious, but a couple of things complicate it.

if its HPA targets CPU utilization, lowering the CPU request can cause it to scale up even though actual CPU usage hasn’t changed.

and after an OOMKill, current memory usage can look fine while missing what caused the previous container to get killed. especially if you’re lowering memory limits along with requests.

both came up while we were building rightsizing into Radar (open source).

for those doing this in prod, what other cases make you think twice before lowering requests?

Hello !

We are planning on handling helmcharts we are currently deploying with Terraform, to do it with Helmfile

How do you usually handle passing secrets and values that are generated via Terraform to helmfile ?

I tried doing a rather simple deployment where terraform creates a secret with all the sensitive values that are then referenced via an externalSecret in the values

I also output Terraform values to a values.yaml file (that helmfile would read) for the rest of the dynamic values.

I was wondering if there was other solutions out there for interfacing Terraform <> Helm(file) values

Thanks

EDIT:

Thanks all for your answers. I know that terraform is not the right tool for helmcharts and that's why we want to get out of it (at least helmcharts)

Argo (or Flux) is what I want to target, ESO for passing secrets from infra that have been created in TF.

A link https://github.com/gitops-bridge-dev/gitops-bridge to describe the interfacing in our case that we aim to (thanks u/bluku14)

Cheers

I'm savaging multiple servers with dual 1080Ti's and 2080Ti's. Most workload VRAM requirements exceed the GPUs.

I was thinking of 2 hypothetical solutions:

- Aggregate GPUs across nodes either into a pool, or if possible a big virtual GPU

- Offload the rarely used pieces of memory in the VRAM to the CPU RAM

Didn't find too much information about any of these, most GPU related topics are about slicing them to fit more Pods, but this would be the opposite.
I'm open to any other solutions to make use of these nodes with older architecture GPUs.

Thank you!

It has been some time since I started working with Kubernetes at scale, from designing, deploying, and maintenance, and time taught me it's a fantastic technology, but also very taxing in the short, mid and long run. One of the things that bothered me most is the 3-node tax and how hard it is to operate at scale.
At some point I decided to shake the monster's hand by reading the source code. What I found: it is just an application with state, nothing more and nothing less, and the software industry has been dealing with stateful applications since before I was born, without ever needing ceremonies to operate them.
In my opinion, the traditional way of operating add-ons can be considered an architectural crime. Various add-ons are installed on clusters to extend functionality, but that extension is a consequence of the primary objective: running workloads. Operators work with a control plane and a data plane, and the problem is both layers compete for the same resources despite not sharing the same availability requirements. In multi-cluster environments, which is the reality for most organizations, the redundancy in resource use and maintenance becomes hard to justify.

To resolve these issues, I built Heir (read why it's called that here: Why Heir?).

Heir is a Kubernetes distribution built around one idea: every cluster's control plane automatically inherits the availability of a management cluster. Rather than dedicated per-cluster infrastructure, Heir runs the control plane as a workload inside a shared management cluster, so a single, well-operated cluster serves as the foundation for as many tenant clusters as you need. Heir ships as a single self-contained binary weighing only 135MB. Architecture: https://tardigradeproj.github.io/docs/architecture

Heir is not Kamaji, Kamaji is explicit about scope, it only hosts the control plane, and it expects something else, usually CAPI and an infrastructure provider  to actually turn up machines and join them as worker nodes. Kamaji itself doesn't ship any node-side agent or join flow. Heir does.

Heir is open to contributions of all kinds, whether code, documentation, or issues. All contributions are welcome.
You can try it by following Quick start guide: https://tardigradeproj.github.io/docs/installation/quickstart

Share any new Kubernetes tools, UIs, or related projects!