That's great for you, but it gives you no excuse to make other people uncomfortable just because you don't think a joke is too far. And no, saying "it's just a joke" isn't an apology.
And that's where context matters! Know when to and when to not joke is key. But the subject doesn't matter IMO. Also bullying 'jokingly' is not joking. It's bullying. But true jokes can be applied on every subject.
Context matters though so you should not be doing it all the time or to everyone. But for me personally, almost no joke is too far.