Most of the time, if people want to physically hurt you, you're either doing something really bad, or really good. I don't think Yelp falls into the "really good" category.
Or you might just be hurting business of bad people. Not every owner of the restaurant is a nice and smart person who can accept negative feedback (aka. bad review) as a sing that he/she needs to change something.
