How in-depth have you used these 4.7-5 inch phones? I ask because it doesn't really map to my experience at all--I have small hands (8" stretch, 6.5" length) and can reach corner of a Galaxy Nexus, Nexus 4, or SGS3 without any difficulty whatsoever.
Regardless of the stretch, my biggest nag is the form factor in my pocket. I don't have a handbag/manbag so all my stuff usually comes with me in my pocket. I also use my phone a lot for music, so a big phone in the gym is a nuisance. I also use it as my everyday camera - I have my phone and an SLR. When I am doing sports activities (biking or snowboarding mainly) I will take my phone. That slightly bigger phone is an annoyance, and I feel as if I am more likely to break the phone. Just a personal preference, but I may have to jump over to an HTC One or a Sony Xperia Z because the processing power is just too tempting to miss out on. I'll keep my old phone as a backup for those other times.