I can see that could be an issue. I guess one solution is to make sure users understand that that is going to happen.

These days everybody is used to their smartphone asking things like "Do you want to allow App-X to use your microphone" etc.