The problem is not setting up an email server; the problem is in ensuring you email is delivered/visible. I set up an email server vps with my domain and domain keys etc but my test emails to my family ,who use Yahaoo and Gmail, would not get delivered. I gave up !
Deliverability is more of a problem than if you use something like Gmail (it's easy for them when they control both ends of the email!), but I've been running my email off of Mail-in-a-Box for more than a year and those sorts of issues haven't been more than a rare inconvenience. [I'm the guy behind the project.]
I have another problem.. my ISP blocks incoming traffic on smtp. For no good reason. I'd have to upgrade to a business class connection ($$$). Unfiltered end to end connectivity would be nice. Preferrably on a symmetric line. I'd call that Internet access.
I'm not running a mailserver anymore, but I use smtp-as-a-service for things that need to send email notifications.
I've been using mandrillapp.com for over 2 years now, completely free. Other people like mailgun.com. Anytime I need to setup a nas or monitoring system that needs to send out notifications, I create a new api key then add that into whatever device needs to send email.
I also for a brief period used the service as a "smarthost" for a shared mail server (not mine directly). This technically worked fine, but became an issue as end-users would get viruses and start sending spam. However, mandrill would notify you of this issue and you'd see high rejection rates. I could even view the rejected message (privacy concerns aside) to see headers of who was sending them. So while I wouldn't recommend it for a shared server, it would be fine for a personal server.
Setting up my own mail server a couple years ago i had this problem. The "best" solution seemed to be routing through a service like gmail, which defeated the purpose.