Do you have any advice for setting up a test Kafka environment? I'd love to be able setup a lightweight in-memory Kafka for unit-y tests without going through the whole Docker compose rigamarole.
Depending on complexity and specialisation of your use that's what may work for unit testing:
1) Abstract real library with tech-agnostic interface to remove as much tech details as possible and focus on behaviour.
2) Create simplest in-memory dummy implementation for that interface, basically a mock.
3) Write a test spec for that interface behaviour (does xxx on yyy, returns zzz otherwise) and run it agains both real wrapped interface & mock to be sure they're consistent.
4) Make code to load mock in local testing env
5) Use real kafka when running it on a testing pipeline
Same way you could abstract various things - e.g. once I just abstracted proprietary services and locally flushed everything into local Postgres. You can emulate multiple services there - e.g. database, somewhat-usable queue, etc. And get some extra control nice for tests (e.g. make it to throw errors on your wish or add timeout to simulate net delays). That saved a lot of time for local development for me, without dropping quality.
Check out the Kafka support in Testcontainers: https://www.testcontainers.org/modules/kafka/. It is container-based, but with a very simple-to-use API for spinning up Kafka in tests.
