It happens to me often enough. Maybe it's just if you have friends who are interested in gadgets, but they will usually say things like, "Oh, you got a new phone? Can I see it?", "Oh, you installed a new ROM? Can I see it?", "Oh, I was wondering how this worked on that phone, can I try it on yours?" All of these are generic requests that just involve tapping around a lot to get a feel for the device, and they're all totally valid requests that one wants to fulfill to help his friends experience more stuff.
Also, some people just don't consider their phones private and don't understand why anyone else would. My parents or siblings sometimes want to flip through Gallery to see the newest pictures of my children, friends, or co-workers.
A couple misplaced photos can be quite the liability in a situation like that...
Are people really handing their phones out that often?
I only hand my phone to someone else when:
* I've asked them to take a photo of me.
* They're riding shotgun in my car and need to call a contact or navigate with info readily available on my phone.
Neither situation is risk for people snooping around.