I also find them baffling and frustrating. It seems that senior sysadmins are fond of "But email is so easy", and I'm sure it is... once you already have your battle-scars (I'm finding Buildbot lies in the same category). Another sysadmin friend of mine says that setting up your own proper mailserver should be a rite of passage before you can call yourself a sysadmin.

Edit: it's not so much that it's about configuring Application X, but that there are so many moving parts, from local aliases to firewall fun to DNS entries. Then there's the fun of 'deliverability', greylisting and so forth.