When a consumer group misbehaves I usually end up piecing it together from kafka-consumer-groups.sh --describe, lag numbers and broker logs. Most of the time the cause is one of a handful of things, so I wrote a small open-source CLI (Go, MIT) that checks for them.

Example: a group that stopped, with messages piling up:

$ mqx group diagnose billing
[warning] empty-with-lag: no members, 25 messages waiting
  The group has committed offsets and unconsumed messages but no consumer is running.
  Fix: Start the consumers, or if the group is retired delete it with `mqx group delete billing`

It samples the group a couple of times and reports findings with a suggested fix. The rules today:

• stale members left behind by an unclean shutdown
• more members than partitions
• one group id shared by apps subscribed to different topics
• group stuck rebalancing
• empty group that still has lag
• partitions with no committed offset
• uneven assignments

mqx group watch <group> prints a timeline of state changes with rebalance durations. It handles the classic protocol and KIP-848.

What it is not: a replacement for Kafka UI or AKHQ. It is a terminal tool with no server. It also does peek, publish, offset resets and topic admin, and it speaks RabbitMQ too.

Disclosure: I built most of it with Claude Code, and the repo has integration tests against real brokers (testcontainers).

Repo: https://github.com/Max2535/mqx

Which causes of rebalances or stuck groups have you hit that this list misses? That is what I will add next.

We just published Part 3 of our Zilla Benchmark Series, this time looking at the performance impact of encrypting Kafka records at the gateway, without requiring changes to producers or consumers.

We compared direct Kafka access, TLS passthrough, whole-payload encryption, and structured field encryption using the OpenMessaging Benchmark.

Some highlights:

• Whole-payload encryption: +3.4 ms p50 and +37.9 ms p99 publish latency compared to TLS passthrough.
• Structured field encryption: +14.8 ms p50 and +56.1 ms p99, with only sensitive fields encrypted and the rest of the record remaining accessible.
• High-throughput test: Sustained 60,000 messages/sec with whole-payload encryption and zero publish errors in a separate benchmark.

All four configurations sustained their target throughput during the main 60-minute test.

A big part of this comes down to Zilla's streaming-native architecture: zero-copy processing, minimal allocations, cached encryption keys, and shared-nothing workers.

We've also included the complete benchmark setup, configurations, tooling, and instructions to reproduce the results so you can run the tests yourself.

Full write-up: https://www.aklivity.io/post/encryption-and-structured-field-encryption-aklivity-zilla-benchmark-series-part-3

I kept switching between kcat, the kafka-*.sh scripts and a web UI during incidents, so I built a TUI that covers the things I actually do: follow a topic live, seek by offset or time, filter (regex, fields, headers, partitions, time ranges), decode Avro/Protobuf/JSON Schema through the registry, watch consumer lag, and reset offsets, with a preview of exactly what changes before anything is written. Profiles are read-only by default.

Works with plain Kafka, Confluent Cloud and MSK IAM. Single static binary for Linux (x86_64/arm64) and macOS. kaffy --demo runs against a fake cluster, so you can try it without Kafka.

GitHub: https://github.com/prasenjithiwale/kaffy

Feedback is very welcome, especially on what you'd need before running it against production.

https://reddit.com/link/1x4tkam/video/0r52p5w7s7vh1/player

Hi Guys - im the maintainer of StreamSkope - https://github.com/asadarafat/streamskope, an opensource workbench multiple platfom selection - desktop(available as unsigned installer) and self-hosted browser(available as a released container) versions.
The focus of StreamSkope is practical Kafka Troubleshoting, finding relevant records, comparing payloads, following corellation identified accross selected topics.

Current capabilities include:
- Saved connection profile with Oauth, PLAIN, SCRAM and mTLS.
- JSON, Avro, protobuf inspection are supported.
- Time/offset searches, filters, saved views, recorded bookmark
- Consumer lag inspection
- Message comparsison

This is an early project, so im looking for feedback from engineers who regulary investigate Kafka data and/or consumer problems.

If you try it, I do particularly like to hear:
- Could you connect and find the record(s) you need? Where you get stuck?
- Which troubleshooting task is still awkward in your current tooling? (i hope i can accomodate it in StreamSkope)
- What would you prevent you from using StreamSkope?

Please share reproducible steps or redacted examples instead of credential or private payloads - obviously.
I will follow the theread and answer questions.

If you found a bug or have a feature request? Please open Github issue, including the app version, OS and reproduible relevant steps. Again please exclude credentials and private payloads.

Contribution and pull-request are welcome, and if you find StreamSkope useful a GitHub star would be appreciated.

I wrote this message post myself. AI tools were used for StreamSkope implementation and the demo video, with me directing the requirements, architecture and iterative improvements.

Hi everyone,

I’ve open-sourced Kafka Connect Healer, a small Go service that polls Kafka Connect’s REST API and requests restarts for failed connectors and tasks.

The goal is to automate routine recovery, with configurable backoff to keep persistent failures from turning into constant restart attempts. Intentionally paused or stopped connectors are left alone.

Some of the features:

• Supports multiple Kafka Connect clusters, with independent restart history.
• Configurable task restarts, polling intervals, timeouts, and exponential backoff.
• YAML configuration with separate credential overlays.
• An HTTP API for inspecting and updating configuration and cluster definitions.
• Structured logs for restart attempts and observed recovery.

You can run it as a standalone binary or build a Docker image. The repo includes Kubernetes examples, configuration documentation, and integration tests against real Kafka and Connect containers covering failures, authentication, backoff, and shutdown.

It’s licensed under Apache 2.0 and maintained by Entropic Works.

GitHub: Entropic-Works/kafka-connect-healer

Docker Hub: https://hub.docker.com/r/entropicworks/kafka-connect-healer

I’d appreciate feedback from people operating Kafka Connect—especially around recovery behavior, deployment needs, and failure scenarios you’d want covered. Issues and contributions are welcome.

In a recent interview I was asked this question:

Assume we have a car leasing platform, you have a post of a car with car details like price, color etc, a post has two buttons, “purchase”, and “more info”, once you click a button “Purchase” a modal window opens, you fill out your details like name, email and phone number, agree to terms and can either submit or cancel your application. Once you submit the application it gets sent via REST API POST request to external legacy CRM system for the manager to contact the client and proceed with the application. The catch is the CRM you’re sending the application to is very old and is constantly not available (can be down for hours quite often). So their question was, how would you design the app so the applications don’t get lost.

I thought it’s quite a simple question (I designed a lot of much much harder things in the past) and
approach was:
Never talk to the CRM directly, send the application to our own backend, save it to database, then periodically send all applications to the CRM.

But after I said this, the interviewers asked well what are the other ways you could do it. And I already knew where this is going, there’s a common cargo cult with the Kafka going on in local tech scene. So I say: well, we could use kafka for the asynchronous integration, but it would not be suitable in this case because it would add unnecessary overhead, we would need to have producers, kafka itself, and the consumers. Mind you, no other input details mentioned, I tried asking for more details they said let’s just keep it simple. So you literally would be spinning whole kafka setup for just one message. After I said I don’t see kafka being the correct solution here, both interviewers confronted why am I so much against kafka. I replied I’m not against it overall, I just don’t see it being the right tool here. Then they asked, well what if there’s quite a big load, like 5000 applications daily. I replied 5000 requests per day is like not even 1 RPS, far from it. Then they said kafka wouldn’t be over-engineering here and it’s the right approach, and my approach was actually harder, I regret not confronting them back though.

They positioned themselves as seniors.

After a couple of these questions they ended the interview and said they were looking for real senior candidates and my level is far from it. They told me to brush up on my integration knowledge as well.

What do you think? Was I wrong here? I don’t think I was, I was quite right in my opinion

Hi,

One of the nice things about our Kafka Simulator is that the cluster configuration and the current simulation state are stored in the URL.

So if you're investigating a particular scenario, you can just copy the link and share it with someone. They'll see exactly what you're looking at.

We recently made this a bit more useful. We published the URL format specification and released a new project, monedula-sim-link. It can read the configuration of a real Kafka cluster and generate a simulator link from it. It also comes with an MCP server, making it possible to work with simulator scenarios through AI tools.

And while we were at it, we reworked the documentation for all our Monedula Flock projects.

URL documentation: https://monedula.dev/flock/docs/kafka-simulator/reference/url-format/

monedula-sim-link: https://monedula.dev/flock/docs/sim-link/

other projects: https://monedula.dev/flock/

Hi r/apachekafka,

I have been building small streaming projects for my blog, and every one of them started with the same Compose file. I turned that into a CLI, odctl, and it is now at 1.0.

bash uv tool install odctl odctl up kafka-lite --dry-run # shows what it would start, in order odctl up kafka-lite odctl down --all

You get a KRaft broker on 9092 (kafka-full runs three), Karapace on 8081, Kafka UI on 8086 and Kafka Connect on 8083. Connect already has the Debezium PostgreSQL source, the Iceberg sink, the ClickHouse sink, the JDBC and S3 sinks and the MSK data generator, so you can POST a connector config immediately.

If you add postgres, it comes up with logical replication enabled and a publication ready for Debezium. If you add catalog, the Iceberg sink writes tables that Flink, Spark and Trino can read.

Two examples I built on it: CDC from a simulated shop with Debezium, and Flink SQL leaderboards fed from Kafka.

It is meant for local development and demos, not production. The Kafka guide shows the basics.

• Kafka guide: https://jaehyeon.me/odctl/latest/guides/kafka/
• GitHub: https://github.com/jaehyeon-kim/odctl

If you try it, please tell me what fails on your machine. Most of my testing has been on macOS and GitHub's Linux runners.

AI: I started odctl without AI. Since July I have used Claude Code for parts of the code, the tests and the docs.

Hey folks —

I’m the maintainer of Klag (klag.dev) — a Kafka consumer lag exporter that started as a maintained alternative after kafka-lag-exporter was archived.

I’ve been really glad to hear from people running it in production, and I’d love more of the community involved in shaping it. If you’ve been wanting a small, friendly OSS entry point in the Kafka space, I’d be happy to help you get unstuck — whether that’s picking a first issue, reviewing a PR, or talking through a feature idea.

There’s a curated board of good-first issues and open discussions for features/devs here:
https://github.com/themoah/klag/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22

Happy to answer questions in the comments.

I used to do this regularly at work: create a topic, let one application write to it, let another read from it, and grant access to its consumer group.

Kafka’s ACL CLI has a --producer convenience option, but it also grants CREATE. In my case, an administrator creates the topic; the producer only needs to write to it. Running separate ACL commands works fine, but repeating them gets old.

So I put together rekaf, a small Bash wrapper around the Kafka CLI:

./rekaf.sh \
  --topic orders \
  --producer orders-writer \
  --consumer orders-reader \
  --group orders-group

It runs with administrative credentials, creates the topic if it is missing, and adds these ACLs if they are missing:

• Topic WRITE for the producer
• Topic READ for the consumer
• Group READ for the consumer

Existing permissions are left in place.

The code was written with AI, and I tested it manually on a small Kafka 4.0.0 setup: topic and ACL creation, repeat runs, producing and consuming as separate users, and failure with an incorrect admin password.

That testing caught a real mistake: --allow-hosts instead of --allow-host. The automated tests had missed it because their CLI stub repeated the same mistake. Fixed it, then reran the checks against Kafka.

It’s a small tool tested on a small setup. Please try it on your own test environment before using it elsewhere.

MIT license, with documentation in English and Russian:
https://github.com/Thoroughly-Pizzled-Studio/rekaf

If you spot a problem with the ACL checks or have a case the script handles badly, I’d like to hear about it.

Disclosure: AI also helped me write this post in English.

It may be well known that losing quorum (2 out of 3 controllers) is not supported, and that bad things happen would entail if it ever happened. But did you ever wonder what to do if that happens? I have published a blog post about how we handle this disaster scenario at Aiven.

Synopsis: this is relevant to anyone interested in operating a KRaft-backed Kafka cluster, and deals with the practicalities of a rare disaster scenario.

Is there a Kubernetes operator for Kafka that watches consumer pods and handles partition assignment and rebalancing itself, instead of Kafka's group coordinator?

The idea: when a consumer pod crashes, K8s knows within about a second, while Kafka waits for session.timeout.ms (45s by default) before rebalancing. An operator watching the pods could reassign partitions right away, and do the same when a new pod becomes Ready.

Does something like this exist, or has anyone built it? If not, is there a good reason why nobody does it?

Hi everyone, have written 2 chapters on Kafka, Will be helpful for someone trying to start with Kafka

Let me know your feedback

Our most fun (and probably nerdiest) Kafka Simulator release so far. 😄

v1.8 adds a terminal emulator with 16 Apache Kafka tools and a few familiar shell commands.

You can SSH between brokers and clients, check consumer lag, change configuration, and inspect the simulated filesystem. Run ls on a leader and a throttled follower, and you’ll see their log segments diverge.

The terminal and the visual simulator share the same state. Commands change what you see on screen. Pause or rewind the simulation, and the CLI lets you inspect that exact moment.

We also added 20 scenarios covering multi-DC setups, disaster recovery and terminal workflows.

https://medium.com/@fhussonnois/kafka-topic-as-a-service-automating-gitops-workflows-with-kestra-and-jikkou-f9eff8633a86

From manual CLI commands to a fully automated, human-in-the-loop provisioning service for Apache Kafka with open-source solutions Kestra and Jikkou on Aiven’s free Kafka tier.

Hello! I'm a big fan of Kafka, like I think many of you are, BUT I've always steered clear of using Kafka for any personal projects because running and maintaining a Kafka cluster for rinky dink project #23 just isn't feasible or smart.

That's why I made SQLStreams. It's an open source, pure Go / SQL library that uses a Postgres Database to give you Kafka-like functionality. It has features like: independent consumer groups, replay, retention and compaction.

I've also made it so each Topic Stream automatically creates a separate retry queue, which seconds as a dead letter queue, because I always found it annoying trying to get retry topics setup and working in Kafka.

If you want a visual understanding of how things work - you can play around with this PGlite demo on the docsite (demo doesn't work on mobile, sorry!).

If you want to run SQLStreams locally you can:

Run Postgres

docker run --name my-postgres \
  -e POSTGRES_USER=username \
  -e POSTGRES_PASSWORD=password \ 
  -e POSTGRES_DB=database \
  -p 5432:5432 \
  -v pgdata:/var/lib/postgresql \
  postgres:18

Install the library / run your code

pool = sqlstreams.NewPostgresPool("username", "password", "localhost", "database") 
client = sqlstreams.NewClient(pool)

stream = client.Stream("videos") 
stream.Register()

producer = stream.Producer() 
producer.Register()

producer.Produce({VideoId: "video-42"})

Note: I've removed some of the extra Go-ness in above code for brevity and clarity. The docsite Quickstart is a more real example if you are interested.

All in all, I'm hoping SQLStreams can be a kind of gateway drug for small to medium sized projects. So they can take advantage of all the great event-driven features of Kafka but without the hassle.

There is still a ton more things to do (implementing circuit breakers for example) but let me know what you think.

And if you've made it all the way here, I appreciate it!

Github: https://github.com/allegedlyreliable/sqlstreams
Docsite: https://sqlstreams.io

Edit: fixed formatting

Today, StreamNative has open-sourced Ursa, the storage engine behind Kafka and Pulsar services on StreamNative Cloud, Ursa for Kafka (UFK), their Kafka distribution that brings diskless topics, and the Lakestream API & specification, an open foundation for building interoperable stream storage on object storage.

In the streaming space, we're seeing a unification of data streams and lakehouse tables. So, the same storage layer can be shared by different protocols and systems without locking data behind a broker. It’s all about building an open ecosystem around stream storage for Kafka, Pulsar, lakehouses, and Streamhouses.

Open source is the right way, whether it's data infra or AI. I also believe that StreamNative has all the strategic components for the next-generation data streaming platform: Kafka service, object-storage-native design for low cost, lakehouse (Iceberg and Delta Lake) support, and the SQL capability to query streams and tables using Postgres-style SQL with RisingWave.

The closest projects I have seen do this are Confluent TableFlow (proprietary) and Apache Fluss (heavily dependent on Flink and RocksDB, and with no Kafka support), where the topic/stream is the table.

An open source alternative:

https://minkdb.com is a Rust native, single-copy streaming database, built on S3-compatible Object Store (a stream is both the table and the topic). Supports log/append-only tables and primary-key tables (upsert, changelog, lookup) over Kafka or Arrow Flight, with history tiered into Apache Iceberg and read alongside the live tail as a single table, and SQL via Flight SQL on Apache DataFusion.

The core of the design is AutoMQ's S3Stream (WAL) storage engine and the table model of Apache Fluss.