We live more on the preventive side where we tap into the real-time data flow before it reaches logs or data stores. Therefore, enabling you to be preventive vs reactive.
This is definitely a valid approach and it is an expensive path. Athena charges a hefty amount per query if you have a large dataset. In addition, this approach won't work if your data is serialized with something like protobuf.
Batch is betting that more companies are going to be utilizing event sourcing in order to scale. We want to be a foundation piece in their data infrastructure and support their transition into event sourcing by initially offering replays. We want to be a "One stop shop" for all event sourcing needs.
Cool! I don't have much data on how many companies are using events for key workflows, but I do know that many, many companies would _love_ to replay HTTP requests!
From their website: Kafka is an open-source distributed event streaming platform.
There are many use cases from piping website activity tracking, metrics, log aggregation and stream processing. For us, it's a communication layer utilized by our microservices. An event goes into the stream and any services that cares about that data will consume it. In other words, it's like an ultra-resilient, scalable, redis pub-sub with history that runs on the JVM. You can read more about the use cases here: https://kafka.apache.org/uses
edit: Sidenote, Kafka is often waaaaaay overkill - if you need messaging, use something simpler like Rabbit or NATS or Redis and only use Kafka if you know why you need it.
Thanks. So an event that goes in would be something like, “user logged in,” and services that care about that data would be...? Sorry still having some trouble understanding it.
Pretty much. A good example might be an online store. Let's say one of your internal services deals with notifying UPS, FedEx, or DHL to pick up and deliver a package from your warehouse and ship to a customer. You could use something like Kafka to store messages about delivers which your internal service will pickup and process and then notify the delivery companies API.
Something like Batch could be helpful in this situation. For example, let's say a dev makes a deploy that breaks only the FedEx delivery notification or the FedEx API breaks in a way your were not expecting. Once the issue is fixed on the dev side or FEDEX side you could use Batch to search for all FEDEX delivers that were handled improperly during the time frame of the issue. This way you are not randomly resending messages to all your delivery companies for an issue that was only related to one vendor.
You might have an achievement service. It watches for logins and grants a user an achievement after logging in for N consecutive days. Your authentication code need not know that an achievement service exists.
We are planning to support as many messaging systems as we can. We will definitely investigate Pulsar. Going to add it to our feature list and make an issue on plumber to support introspection on Pulsar. Cheers!
Good to know that Pulsar is on your roadmap. Also, kudos to see user-land tooling around a common painpoint for teams doing any event processing at scale.
Thank you! We felt the pain point while actively trying to build observability tools in order to debug our messaging systems. We built plumber to standardize some of our internal tools and then decided to open source it to help others who are feeling the pain.
