Problem state

Every team in my company started building its own AI agent and they were getting more out of control. Most use cases were similar: code review or doc Q&A, the kind of thing an general agent handles fine. The bigger issue was not the count, it was the state. What is that agent even doing? Which skills got installed on it, by whom? All of that lived in config files inside pods. So I've decided to build a platform to address it.

Components

This platform consist of two components

• Web: User-facing web server. It manages Hermes agents.
• Custom resource operator: Reconcile a HermesAgent custom resource to manage Hermes agents.

State outside the pod

The general agent we standardized on is Hermes. It is a personal agent: config.yaml plus skills, plugins, and cron jobs installed via commands, all stored in local files. Powerful for personal use, but hard for a team, because state inside a pod is invisible state.

An agent needs config.yaml, secrets, network, storage and etc. Kubernetes already has a mechanism for each, so I decided to declares agent state in a CRD. The operator

• config.yaml comes from a ConfigMap the operator mounts
• skills and plugins are listed by identifier in the spec, and init containers install them before the main container starts
• pip/npm packages pre-install the same way

Ex) HermesAgent custom resource

apiVersion: agents.hermeum.app/v1alpha1
kind: HermesAgent
metadata:
  name: code-review-agent
spec:
  hermes:
    config:
      raw:
        model:
          default: glm-5.3
          provider: ollama-cloud
        platforms:
          webhook:
            extra: ...
    workspace:
      files:
        SOUL.md: You are a pragmatic repository maintainer ...
    storage:
      persistence:
        enabled: true
        size: 5Gi

Now, the agent is declared in a manifest and the operator keeps it in sync. The state of Hermes is configurable from outside the pod.

Hide the infra

The CRD carries infra config too: storage, networking, security. But most users do not want any of it. Hermeum defines a thin agent spec holding only the core settings (config, skills, plugins, crons) plus an agent type field. A mutating admission webhook patches the HermesAgent resource at creation time, filling in the infra fields that match the type. The agent spec no longer touches infrastructure.

Ex) Agent spec

name: Hermeum GitHub Maintainer
description: Maintains hermeum/hermeum on ...
type: small       # <- This is the agent type. The admission server patches the storage and networking fields
soul: You are a pragmatic repository maintainer ...
config:
  model:
    provider: ollama-cloud
    default: glm-5.3
  platforms:
    webhook:
      extra: ...

And the greatest thing of management via manifesto is we can leverage the chatbot to create and manage any Hermes agent we want. The chatbot in the platform write up the manifest so non-dev members are able to create their own agent.

What it left us with

Every agent and its state are visible in one place. We raised agent visibility to tackle the agent sprawl problem.

Questions and criticism welcome. I am specifically curious how others here are handling per-agent config drift.

My 3-node Talos homelab had no persistent storage, so anything with data, like Grafana or a database, had nowhere to live.

Longhorn keeps every volume on two of the three nodes, so one node can go down without losing data, and it rebuilds the missing copy on the third node.

It also takes hourly snapshots, and backups to S3 are optional. The Longhorn UI runs on my LAN and over Tailscale.

Repo: https://github.com/prateekjaindev/homelab

Write-up: https://blog.prateekjain.dev/persistent-storage-for-a-3-node-talos-homelab-with-longhorn-587c05356b73?sk=9093231c2d278ba92fd493098fbe237d

What are you using for storage in your homelab?

I pulled together the current EKS support dates, since a few are coming up soon:

Version |End of standard support |End of extended support
1.31 |Nov 26, 2025 |Nov 26, 2026
1.32 |Mar 23, 2026 |Mar 23, 2027
1.33 |Jul 29, 2026 |Jul 29, 2027
1.34 |Dec 2, 2026 |Dec 2, 2027
1.35 |Mar 27, 2027 |Mar 27, 2028
1.36 |Aug 2, 2027 |Aug 2, 2028
1.37 |Dec 1, 2027 |Dec 1, 2028 What this means:
- Once a version is in extended support, the control plane is $0.60/hr instead of $0.10/hr, about $365/month more per cluster. Running dev/stage/prod on 1.33 is roughly $1,100/month extra for nothing.
- 1.31 is 5 weeks from end of extended support (Nov 26). After that AWS upgrades the control plane for you, which is not a fun surprise if you have deprecated APIs in use.
- 1.34 is the next to tip over, on Dec 2. Upgrading to 1.35 buys you until Mar 27, 2027, and 1.37 (released Oct 1) gets you to Dec 2027.
- You can set the cluster upgrade policy to STANDARD so it won't drop into extended support, but then it auto-upgrades at end of standard support instead.

Before upgrading, the usual: check for removed APIs (pluto, or EKS Upgrade Insights in the console), bump add-ons (VPC CNI, CoreDNS, kube-proxy), and remember node groups can only be a limited number of minor versions behind the control plane.

Disclosure: I built a free tracker with these dates plus RDS, Aurora, Lambda, etc. It refreshes daily from endoflife.date and AWS pricing: https://eolwatch.pages.dev. Happy to fix anything that's wrong.

The pitch for AI SRE tools in here always focuses on alert correlation: taking 50 firing Prometheus alerts and summarizing them in Slack.

But alert storms are almost always just downstream symptoms. When a consumer backs up because someone pushed a bad memory limit in Helm or a secret didn't sync properly, an LLM looking at pod logs and metrics will just confirm that the pod is restarting and the queue is full. It can't diagnose the actual failure because the cause lives in deployment history and manifest drift, not in runtime telemetry.

Automated troubleshooting only becomes useful when it can query recent GitOps diffs, environment baselines, and workload ownership instead of just grepping container logs. Without that context, alert correlation just tells you things are on fire in slightly nicer English.

I built zlb for clusters where:
- public IPs are scarce and routed to one machine;
- nodes have no inbound internet access;
- there's no BGP or shared L2, so MetalLB, Cilium IPAM, and kube-vip don't fit.
A controller watches LoadBalancer Services and pushes the desired state to a small daemon on that machine. The daemon programs nftables DNAT from publicIP:port to node:NodePort, keeping the client IP.
Services are published on their NodePort, so all of them can share one IP: two Postgres Services on 5432 become IP:31543 and IP:30876. Literal ports like 80/443 come from explicit reservations (e.g Ingress Controller gets 80/443 on the public IP).
TCP/UDP/SCTP, Local/Cluster traffic policy, loadBalancerSourceRanges enforced at the edge
Atomic, verified nftables changes with rollback; forwarding survives crashes and upgrades
TLS/mTLS API, sandboxed non-root daemon, Helm chart with optional SSH bootstrap of the VM
Limits: the edge node is a single point of failure for now (active/standby with VRRP and conntrackd is planned). IPv4 only, no metrics yet.
https://github.com/abexamir/zlb (Apache-2.0)
Feedback welcome, especially from anyone running a similar single-entry setup.