Some places have church tax based on your religion, such as Germany. I've heard varying claims from locals about the difficulty of leaving organised religion, though for me as an Auslander it was as easy as just saying "none" in the right box when setting up my tax ID.
How does one officially leave the Catholic Church?
Sure, you can stop attending church and taking communion, but you're still on a list somewhere in your parish(es).