You're right! That's a very real and very reasonable concern.
That said, it's fairly uncommon that the choir actually walks away electorally. It tends to require something very dramatic, like Roy Moore. It's far more common that they grumble a bit and vote for whatever anyway, because you committed the sin of advocating for the policies they want while not centering the FBI's racist history.
Again, your concerns are quite reasonable. It's just perhaps possible that what it takes to lose those votes is more than might be guessed.
