I had the opposite experience. Everyone I tried to communicate with via google translate responded positively except one disgruntled train station attendant.
I would say excuse me, I don't speak Japanese can you please help me. And then I'd show them my phone with the translated version of whatever my query is and then press the mic button and they'd get the point that they can speak now. Worked great.
In my experience the whole thing works better with people typing. Firstly voice recognition isn’t perfect. The person has to check what they say was transcribed correctly and can’t correct it with a keyboard if they see it’s wrong the first time, they have to just try again.
Typing means people go a bit more slowly and think about what they are writing, sometimes reworking sentences to be clearer.
Also if they are typing in front of your face, the translation constantly updates based on the partial sentence they’ve typed and this can be quite revealing, which is useful if the final translation isn’t crystal clear.
For face-to-face it wasn't great because speech recognition is error prone. Usually I would always type and the person I'm interacting with would speak and if they saw the text was wrong they'd run it again.
For chatting over sms/whatever it was great. Enough to have complex conversations and get to know people that I couldn't possibly communicate with otherwise. It's not perfect, but it's a lot better than nothing if you aren't proficient in the local language.
Also your mileage will vary by language. English-Japanese seems to be a pretty good combination. English-Chinese and English-Russian are a lot more flaky.
I would say excuse me, I don't speak Japanese can you please help me. And then I'd show them my phone with the translated version of whatever my query is and then press the mic button and they'd get the point that they can speak now. Worked great.