Warpstream (by Confluent (an IBM company))

For Rust-based Kafka alternatives, I like Tansu[1]. It at least provides Kafka API parity, and critically also gives users a pluggable backend (embedded SQLite, S3 for low cost diskless type workloads and Postgres because just use Postgres)

It’s nice to try and out innovate Kafka, but I fear the network effect can’t be beaten unless the alternative is 10x better.

Something like Warpstream’s architecture[2] had a shot at dethroning Kafka, but critically even they adopted the Kafka API. Sure enough, Apache Kafka introduced a competing feature[3] within two years of warpstreams launch too.

[1] - https://github.com/tansu-io/tansu [2] - https://www.warpstream.com/ [3] - https://topicpartition.io/blog/kip-1150-diskless-topics-in-a...

That is hilarious. The product looks like garbage to me. The business opportunity looks like it can be incredibly lucrative if it does catches on.

Aren't nanoplastics a larger concern?

All microplastics out in the open will degrade to nanoplastics at some point, and those find it much easier to infiltrate the human body. They penetrate the blood-brain barrier.

This leads me to believe that it's literally impossible to avoid. The air/water supply must be getting more poluted by the minute with these things

I wonder if plastics, being non-polar, might accumulate in organs which are a bit on the "oily" side - the liver, adipose tissue, and the brain.

how is node failure handled? is this using KRaft or ZK?

These are real trade offs!

Client impl. can be re-created once in a library/extension and be done with.

The network effect of Kafka is the hardest to topple. The API is standard. If you need that connectivity, then use Kafka.

An alternative idea I did not mention is to use a Kafka API proxy on top of Postgres. Tansu is developing this (amongst other pluggable backends) - https://github.com/tansu-io/tansu; That solution would retain both the client implementations and tools (network effect). But it comes at extra complexity

Exactly. "a properly-configured Kafka cluster" implies you have very properly configured your clients too, which is almost never the case because it's practically very hard to do in the messy reality of a large-scale organization.

Even if you somehow get everyone to follow best-practices, you most likely still won't get to saturate the network on "minimal hardware". The number of client connections and requests per second will likely saturate your "minimal CPU".

It's true that minimal hardware on Kafka can saturate the network, but this mostly happens in low-digit client scenarios. In practice, orgs pushing serious data have serious client counts.

Can you give some examples? I'm super curious about single-node Kafka use cases in general

I believe most setups using DB+{Queue,Kafka} don't truly deal with it fwiw.

Agree but we really have to put a number on baseline traffic and max traffic burst in order to be productive in the discussion. I would argue that the majority of use cases never need to be designed for a max-traffic-number that PG can't handle