Interesting, I didn't realize "expectation of privacy" was related to (and dependent on) the 4th Amendment. Seems like a gap in our laws; expectation of privacy from people who aren't the government.
I mean, society won't function if you have to give everyone outside of your home consent to look or speak at you. Unless you plan on blinding and deafening the whole world, there's going to have to be a line somewhere about what we are allowed to learn about the people around us.