Depending on complexity and specialisation of your use that's what may work for unit testing:
1) Abstract real library with tech-agnostic interface to remove as much tech details as possible and focus on behaviour.
2) Create simplest in-memory dummy implementation for that interface, basically a mock.
3) Write a test spec for that interface behaviour (does xxx on yyy, returns zzz otherwise) and run it agains both real wrapped interface & mock to be sure they're consistent.
4) Make code to load mock in local testing env
5) Use real kafka when running it on a testing pipeline
Same way you could abstract various things - e.g. once I just abstracted proprietary services and locally flushed everything into local Postgres. You can emulate multiple services there - e.g. database, somewhat-usable queue, etc. And get some extra control nice for tests (e.g. make it to throw errors on your wish or add timeout to simulate net delays). That saved a lot of time for local development for me, without dropping quality.
1) Abstract real library with tech-agnostic interface to remove as much tech details as possible and focus on behaviour.
2) Create simplest in-memory dummy implementation for that interface, basically a mock.
3) Write a test spec for that interface behaviour (does xxx on yyy, returns zzz otherwise) and run it agains both real wrapped interface & mock to be sure they're consistent.
4) Make code to load mock in local testing env
5) Use real kafka when running it on a testing pipeline
Same way you could abstract various things - e.g. once I just abstracted proprietary services and locally flushed everything into local Postgres. You can emulate multiple services there - e.g. database, somewhat-usable queue, etc. And get some extra control nice for tests (e.g. make it to throw errors on your wish or add timeout to simulate net delays). That saved a lot of time for local development for me, without dropping quality.