It's rude to call anyone something they aren't, doesn't matter where from.

And Germans tend to not be the issue, the people I've met who I'd throw that word at very sincerely tended to be from Austria or Hungary, not Germans.