Getting the wrong case for an article or picking the wrong conjugated form for the verb is about as far from being a "grave grammatical mistake" as can be. Yes it matters, no it's not remotely reasonable or acceptable to just stop listening to someone because they said "dem" instead or "der", or the fucked up adjective declension (which is nice to get right but objectively sometimes carries information) for example.
And no we really don't care, or at least I don't. I am a native English speaker who goes months at a time without speaking to another native English speaker. I would never judge someone for it. I would be ashamed of myself to just stop listening to someone for mixing up word order slightly or dropping a preposition or picking the wrong conjugated form of a verb. If I understand them, then it's fine. If I can help them improve their English, by pointing something out, I will. It's because I'm not an arsehole. Ignoring someone as soon as they mess up a verb conjugation is honestly detestable.
We are talking here about using wrong grammatical gender throughout your speech. That is a bunch of very noticeable mistakes that affect multiple words in the sentence. Just because English does not have grammatical gender does not mean it does not matter in German.
The difference between "der and dem" is NOT gender mistake. And it is also less noticeable as treating the masculine word famine, due to the way it affects other words.
I know it's not a gender mistake, but a mistake in the case is generally more meaningful than a mistake in the gender. A mistake in gender is even less important. I agree one should try. I certainly try. I can often look at a word I've never seen before and "somehow" know the gender, but it is ridiculous to just ignore someone who makes mistakes. And it makes no difference what the nature of the mistake is, grammatical gender isn't special, it's not acceptable in English to treat someone like shit because they used the wrong conjugated form of "to be", it shouldn't be acceptable to treat someone like shit for using the wrong gender in German.
English doesn't have gender or anything but a vestige of a case system, but it's still just as complex a language as German. Your comment that you wrote right there is simply incorrect English at worst and unidiomatic at best, and you made a number of grammatical mistakes, but I wouldn't think to ignore you for it, because that would be completely ludicrous. In fact I'm only even mentioning it because of the conversation topic at hand. And if you're making so many mistakes whilst writing English then I'm certain you're making loads whilst speaking it. Still wouldn't ignore you or discount what you're saying because of it. In fact I wouldn't care at all because I understand that learning a language is not easy.
And no we really don't care, or at least I don't. I am a native English speaker who goes months at a time without speaking to another native English speaker. I would never judge someone for it. I would be ashamed of myself to just stop listening to someone for mixing up word order slightly or dropping a preposition or picking the wrong conjugated form of a verb. If I understand them, then it's fine. If I can help them improve their English, by pointing something out, I will. It's because I'm not an arsehole. Ignoring someone as soon as they mess up a verb conjugation is honestly detestable.