Maybe for some people? Most of my buddies are of the opposite gender, and me hanging out with one of them doesn’t imply anything different to me hanging out with someone of my own gender.
There are definitely people it matters to, I sometimes get tired of the usual remarks and end up awkwardly using "they" or phrasing tricks to try to avoid them
