I also self host but on a VPS. Incoming mail works great. Outgoing mail not so much unless I send it to a trusted third party SMTP server. No outgoing mail fees if low traffic. Might be an option to consider while building up reputation?
I had all that setup from day one and still ended up in the GMail spam and banned by Microsoft/Outlook.com (which also includes anyone using Office 365 in their workplace for mail)
I assume that's because I had no reputation, although I did start on a DigitalOcean VPS before I learnt about their terrible ongoing reputation for ignore abuse reports. [1]
The two typical choices are rspamd and amavisd/SpamAssassin, at least when you're using Postfix on a unity environment. I use amavisd on my primary and rspamd on the secondary, both seem to work OK.
Yes, spam will still make it through and you have to train the filters in either case.
