It remains a constant frustration to me that Postfix, Exim, Cyrus, Dovecot and the like still feel as if they belong very much in the "here be dragons" territories of the Unix world. Configuring these systems is an exercise in constant frustration and bafflement.
They're such a pain to use that since becoming the sysadmin in charge of our work email servers, I gave up running personal ones as well and just pay FastMail to deal with it for me - ain't nobody got time for that.
I often dream of taking a sabbatical and writing modern, user-friendly SMPT and IMAP[0] servers. If only so I could use them at work on my return and save myself a lot of time and stress.
It is very much "here be dragons" but a lot of that is because of legacy compatibility issues from when mail was delivered directly to local users.
If you don't care about local access (IMAP & POP is good enough, no mutt to the mail spool for you) Dovecot is a huge improvement on everything else. You configure it to listen directly for LMTP and use MySQL for user information and Maildir for data it's almost as easy as running a random php+mysql webapp.
Even serverside filtering with sieve "just works".
The inbound SMTP/Spam filtering stack is still a PITA but that's because of security issues (spam).
DBmail (dbmail.org) is one rework that seemed sane but stores all messages in a RDBMS and I didn't want to deal with scaling it at work (little ISP, ~10k mail users, ~1.5 million messages per day including spam) but it was fine for personal use.
I've been playing with homegrown POP & SMTP servers that use a s3 compatible datastore as the backend but that's a side project. People go crazy over email so I really want a simple to operate, sane, zero point of failure mail system...
Re "zero point of failure mail system": I also use Dovecot, and I learnt about DSync recently, although haven't played with it yet. It provides two way synchronisation of mailboxes between two Dovecot servers, so you can store the same mail on two boxes, in completely different locations.
The clever thing is, it can recover from a split brain scenario completely, safely, and without any losses. If your two servers can't see each other for a few hours and you make conflicting changes on both of them, then it apparently is able to recover completely and entirely automatically when the connection comes back up.
You could have one remote server in a DC, and the other locally in your office. Point your mail clients at the one in the DC, but then configure your office router to intercept connections to the DC server and re-route them to the local office server. When you're in the office, you hit the local office server. When you're outside of the office, you hit the DC server.
[edit] Any chance your can elaborate on your S3 test setup? I've considered something similar. I would be interested if there was any mail server software that already does this. The alternative would be to use an S3 based filesystem as your store, but this doesn't seem very efficient. You'd definitely want local caching of messages in this setup.
So, for S3, I cheat and have my own Riak CS cluster so storage is fairly quick and cheap. I should probably use Riak unless the message is too big and then put the body in CS but I was thinking that straight-up S3 is a good lowest common denominator for storing things.
Basically when I started out learning Go I saw a presentation by bradfitz where he made a POP3 server that used twitter for the backend and thought hey, access S3 instead. He had a smtp library on github so I hacked up something that accepts all mail and stores it in a S3 bucket too. I haven't hacked in things like user auth so it's all very proof of concept/useless at this point.
I'll have to take another look at DSync. The SSH batch job per user scared me off last time. I might be able to use it for HA/Failover.
There's probably something wrong with me but I quite enjoy the masochistic rituals of getting postfix/dovecot/opendkim/spamassassin/zarafa humming.
Postfix is actually a beautiful piece of software once you spend some time with it and get to know it. Like most things, once you've done it a few times (and written everything down!) it isn't so bad :)
LDAP on the other hand... that's something that _always_ defeats me.
>> Postfix is actually a beautiful piece of software
Couldn't agree more.
I've also learned a ton from the source code. If anyone has even a passing interest in security, i'd highly recommend browsing the postfix source code.
Designs of much more elegance have been explored (qmail). But as a matter of fact, neither postfix, nor qmail, nor exim, nor (god forbid) sendmail belong into our day and age anymore (I have run 3 of them at scale).
A modern MTA is way overdue. Please keep writing MTAs kids!
In modern languages. Until you get one right. Thanks!
Sorry moe but yes, just yes. It's rock solid, battle hardened, has a fantastic security record, is extremely resource efficient, has gobs of up to date and useful documentation, an active mailing list, a primary author who's stuck with it since conception (unlike DJB) and offers enough flexibility to reliably support virtually any mail setup you care to dream up.
I can't see why in the world anyone would advocate throwing all of that in the bin and adopting something new, flashy and unproven that happens to be written in go/rust/node/other language du jour? I mean why? So we can write configs in JSON and get a 'free batteries included' web management console for monitoring queue lengths?
So we can write configs in JSON and get a 'free batteries included' web management console for monitoring queue lengths?
Yes. Exactly that.
Postfix requires the user to understand pretty much all of its delicate (and often very much ass-backwards) abstractions at once, to accomplish even the most trivial tasks.
Aliases vs virt aliases, transports, the six dozen magic interactions of $mydestination, $myorigin etc. with one another and everything else, ... don't even get me started...
Nothing is easy in postfix. Everything is deeply entangled, counter-intuitive and error-prone. And there is absolutely no reason it has to be that way.
See what elasticsearch did for FTS. This is how software is designed today.
I'm not bashing Postfix, it's still a workhorse. But let's not be sentimental about software. Being better than sendmail simply is no longer the big deal that it was in 1998.
Today I expect my MTA to be all that postfix is, and easy to use.
It's interesting that you brought up ES actually as I've just been toying with it as part of a project I'm working on (I usually use Solr). Unfortunately, I hated it so I guess we just have very different views on what constitutes a good approach to software.
Things I didn't like with ES include:
- Awful documentation (even the newly released guide) and too much marketing speak. I constantly felt like I was reading a brochure rather than a man page.
- I found it opaque and difficult to get clear, concise info on how it's put together and what it's doing under the hood.
- Marvel was nice but it's a shame they charge for it (it's not _that_ nice)
- The install routine (in an effort to be super simple I guess) just felt wrong and needlessly over simplified (I didn't like it making decisions without telling me what it was doing).
And more...
I will concede that working with ES's config files is a lot nicer than dealing with Solr's XML though! And in case you're in any doubt, I am by no means an enormous Solr fan either.
Anyway looks like we'll just have to agree to disagree on this one :)
Edit: also, I am occasionally prone to a bit of sentimentality for good, older software (it took me far too long to give up on apache and embrace nginx for example) - so there's definitely a little bias on my part!
Edit 2: If you could get Salvatore Sanfilippo and Igor Sysoev to team up and build a modern MTA, that would interest me :D
Now that's an interesting comment, thanks for that.
I'll probably come across as arrogant now (sorry), but your critique of ES sounds almost ironic to me.
It's especially curious as you do claim experience with SOLR. Have you ever scaled it beyond a single core?
I'll go on a limb and claim that most of what you perceive as opaque in ES is likely magic that actually works. The rarest kind.
Yes, Marvel is not optional to run a meaningfully sized cluster. But where is Marvel for SOLR again? Where is automatic clustering and balancing? Oh, it doesn't exist?
Sure ES isn't perfect. But it's one generation ahead of SOLR. And Postfix is one generation behind SOLR. That's all I'm saying.
I started out with Exim (3.x or 4 -- can't remember now) and stuck with it mostly for the same reasons that Debian didn't change away -- I wasn't convinced postfix was a significantly cleaner/simpler design -- that it would be worth it to learn how the "other" system worked.
I did have some exposure to qmail before that, and it did feel like it was simpler in some good ways -- but then it became a sort of abandonware, sadly.
[edit: I did recently switch a few vps' over to nullmailer rather than use exim4 configured for smarthost delivery -- once I figured out how to get tls working thanks to this blogpost http://metz.gehn.net/2012/11/nullmailer-with-starttls/)]
Are you talking about the interface to the sysadmin - the fact we see the nuts and bolts in master.cf? This is of course an unfortunate side effect of the power and flexibility on offer.
What i was trying to dig at was more about the code than the user facing side of things.
It beautifully modular. It clearly written - that's a crazy hard achievement for software as committed to security as postfix.
It's dead easy to hack on the code - you could arrive and even without looking at the first class documentation you would be able to tackle implementing a change. The source is that thoughtfully laid out.
I've been running it for a few years now, admittedly with a very basic setup. My config is about ten lines long, written in five minutes or so. I haven't really needed to touch it since.
I love OpenSMTPD. The only issue I had with it was, no filter API (just fixed, as of a month ago or so) and to get things like spam and antivirus filtering, or domainkeys, you have to proxy chain it. Also, the API changed very rapidly, and I had to run nightlies for a while to get around a few issues (I'm on stable now though with no problems.) These things add complexity to the configuration, but I would still rather do this than maintain any other mail server software.
It's on my radar, but to date I've not had (or more accurately made) the time to seriously evaluate it. I'm looking at an architecture overhaul soon, so will make the effort to take a proper look before then.
At some level I've always viewed this as an example of how FOSS can fail to deliver good products at times. It should be embarrassing to the FOSS community that something as fundamental as email services requires super human effort to setup, administer and maintain. I don't know how it is we got here. It's a real head shaker.
I experienced something completely different. I set up a postfix/dovecot/ldap/nginx/roundcube/spam/... for a medium sized office with some special needs last month and was amazed how easy it is. Especially postfix and dovecot are a breeze to work with.
The main problem was, that you can do everything in X different ways and have to think careful which way you want to go.
In addition i am running a very simple private setup for years now without any problems.
I also find them baffling and frustrating. It seems that senior sysadmins are fond of "But email is so easy", and I'm sure it is... once you already have your battle-scars (I'm finding Buildbot lies in the same category). Another sysadmin friend of mine says that setting up your own proper mailserver should be a rite of passage before you can call yourself a sysadmin.
Edit: it's not so much that it's about configuring Application X, but that there are so many moving parts, from local aliases to firewall fun to DNS entries. Then there's the fun of 'deliverability', greylisting and so forth.
Well, you only need two from your list. And I don't really think Postfix+Dovecot is much harder than Nginx+$dynamiclanguage. It's just something fewer people felt the need to do historically.
Setting up mail servers is the one things I truly dread. Its truly barbaric. I cuss and froth at the mouth whenever I have to go down there into that postfix dungeon.
Its the reason I got into Puppet and Ansible - vainly hoping all those barbaric problems could be encapsulated.
Do anyone have thoughts/feedback on Haraka - a mail server used by craigslist http://haraka.github.io/ ( Runs with Node.js + Javascript based plugin system )?
I've used Haraka standalone for a couple of custom-jobs, but I've never used it at any significant volume. That said qpsmtpd was a pleasure to use, deploy, and develop against, so I'd expect to have a similar feeling with it.
On a serious note - you'll find the Haraka developers more friendly and active than any other MTA out there. Just join the #haraka IRC channel on freenode and we're happy to help anyone.
I dread having to set up mail servers. Setting up a typical dovecot + postfix install with a webmail frontend that looks as good as gmail should be as easy and simple as setting up wordpress, but it isn't even close.
Part of the problem is that many years ago certain
people decided it would be a good idea to tightly
couple email to domain names (DNS). Previously email
needed only IP addresses to work.
The result is that now when you are configuring SMTP
you have to also configure DNS. That means more things
that can go wrong, and more things to check as you are
setting things up.
It also means you may need to pay a fee for a domain name.
This is because we all submit to the notion of an ICANN root
and commercial registrars selling (renting) names that
cost nothing to create. Thus email is not solely under your
control. You generally have to play the ICANN DNS game, only
because your email recipients are playing. Nothing stops
anyone from running their own root though. And this is what
is done with private DNS inside organizations.
And then, as if that DNS complication was not already enough
to take control of email away from you, you have various
schemes trying to prevent spam that discriminate for or
against mail you send based on IP address and domain name.
Can you operate email without DNS? Technically yes. There
was a time before DNS, and email worked just fine.
Practically speaking, today you need DNS, whether it's under
ICANN's root or your own.
All this hassle steers you to just accept third party
email hosting. Profiting from this arrangement has become
a career for many a man. And with "the cloud" many are
hoping to cash in yet again, as organizations who once ran
controlled own email feel pressured to let a cloud computing
vendor control it for them.
The fact that all this third party control makes
warrantless search and surveillance so easy is but one side
effect. Centralising hundreds and thousands of accounts in
third parties make the spammer's job easier, too. If you
think about it, there are many unwanted side effects of
centralizing email. When every sender and recipient are
connected directly to each other via a network, why would
you want to prevent them from sending messages to each other
directly?
With the constant connectivity and bandwidth we have today
in many places, the centralisation and outsourcing of email
is baffling to me... it is nonsensical... until you remember
how much of a PITA it is setting up email.:)
It's no wonder we let third parties handle it. Is this PITA by design? Who
cares? Let's just fix it. More of these projects should
exist. Or made public (I imagine many of these are personal
setups now being released for public use). I have my own
that uses qmail.
DNS is not the issue with mail and MTAs. Setting up an MX record is something you can do after googling and reading for about ten minutes. I have only anecdotal evidence to prove this, but that's basically how I set up my own first mail server.
What was a lot more difficult was setting up the actual mailserver itself. Even a simple, two-mailbox-operation was an exercise in frustration when it came to trying to get mail working on a little VPS of mine. Shit, you have to make the sendmail config. How balls-out insane is that?
More recently, there's little working tutorials to get yourself a working dovecot/postfix server, which are relatively easy to understand (thanks, digitalocean!) but I just checked out the first one I found on my google search and it's 2,800 words long. 20 pages if you were to print it out dead-tree style.
I can give you a tutorial on DNS and MX records in much less time than it would take to go through setting up any MTA on linux, and that's the trouble.
But I am actually referring to something different:
hosting your own mailserver.
So when I say "set up DNS" I mean set up a DNS server,
not simply an MX record. This allows you to create your
own domain names and hence email addresses. As I said
above, these email addresses are valid so long as you
and the recpipient use the same DNS root (e.g., ICANN's
root in the case of the public internet).
As bad as things are in terms of the relative difficulty
of setup, I think there are defenders of the status quo
for email and I imagine this explains how I could be
downvoted for my comment.
Don't get me wrong, I love email. It is the reliance on
others to handle 100% of it that troubles me. It is purely
a control issue.
No, you said that. I said that having email so closely
coupled to DNS is part if the problem.
I'm not sure why you would have to remember IP addresses.
We routinely "dial" telephone numbers by selecting from a
list of contacts. IP addresses are approximately the same
length as telephone numbers. The folk wisdom is that people
can remember about 7 digits. But even if you disagree on
all of this, what does that have to do with letting someone
else control our email? The issue here is control, not
whether we use names or numbers or something else when we
enter the address of the recipient.
The nonportability of IP addresses is a problem in its own
right, but I don't see the relevance here. Again, you are
trying to engage me in a debate over domain names versus IP
numbers. Perhaps that is an interesting issue, but here
I am interested only in the issue of control over email (and
because email and DNS have been coupled together, DNS).
And that is what the OP is interested in as well.
I said that email and DNS are closely linked and this makes
email more challenging for any user to control. 1. Because
it complicates the setup and 2. because DNS as we currently
accept it is controlled by third parties.
You are trying to suggest that I am advocating against having email addresses that use names instead of numbers. I am
not.
If email is linked to DNS, and someone else controls DNS,
then you cannot control email.
If you disagree with the preceding statement then please
explain.
With every solution that comes along to solve the "host your own email" problem, and every guide that takes you step-by-step, I always find myself just wanting a huge wiki-style site with all the topology of a good, secure email/groupware server explained in plain English rather than a package that purports to do it all for me. Something with breakdowns/discussion about the different options that exist for the different pieces that make up the whole.
It's one thing to easily obtain my own personal mail server, it's another entirely to understand and be able to maintain/improve it.
Very nice. Comes with Roundcube so you've got a webmail client available.
- DKIM signing on outgoing messages (opendkim).
- The machine acts as its own DNS server and is automatically configured for SPF and DKIM (nsd).
I remember many people running their own SMTP server used to have massive issues with their outgoing mail being silently or non-silently ignored or sent to the spam folder. Does this take care of this? "Mostly" really isn't good enough.
> I remember many people running their own SMTP server used to have massive issues with their outgoing mail being silently or non-silently ignored or sent to the spam folder. Does this take care of this? "Mostly" really isn't good enough.
Frankly I've had mixed results over several years of hosting my own personal email server, even with a valid PTR, SPF, and DKIM. Most SMB and personal email systems I interact with are fine, as aside from Bayesian filtering those tend to rely heavily on blacklists which I am responsible enough to keep myself off of—easy given that I'm the server's only user. And I've never had an issue sending to Gmail users.
Microsoft is the real f-up in this regard, originally with Hotmail and now with the updated Outlook.com. Even though I'm on no blacklists and pass both their SPF and DKIM checks:
Authentication-Results: hotmail.com; spf=pass (sender IP is XXX.XXX.XXX.XXX) smtp.mailfrom=redacted@example.com; dkim=pass header.d=example.com; x-hmca=pass header.id=redacted@example.com
X-SID-PRA: redacted@example.com
X-AUTH-Result: PASS
X-SID-Result: PASS
my messages still end up in the Junk folder.
If you visit their support page you're recommended to join their partner Return Path's Sender Score Certified Email program to stop junk filtering of your legitimate messages. That's great, if you're a medium-sized organization that doesn't mind paying their certification fee; but Return Path won't even allow an individual to register (I've tried). I'm familiar with Hanlon's razor, "never attribute to malice that which is adequately explained by stupidity", but this at least feels like a racket.
Unfortunately I don't know what else to do, other than continue to advise everyone I know not to use Outlook.com and friends.
And to finally answer your question, if this VM runs its own DNS then it probably takes care of SPF and DKIM adequately, but you'll likely have to contact your hosting provider for the PTR record, which is at least as important. But even following those best practices you can still have delivery issues such as I have described above, leading to the unfortunate reality that if you really need a message to be delivered you're probably better off sending it through Gmail.
I was suffering the same problem about 7 years ago on my own personal mail server. I discovered that if I padded out the message headers with a load of junk headers to push the message size up beyond about 50KB, then the otherwise exact same message would arrive in the Inbox at Hotmail rather than being blackholed.
I will. One thing I can add is Microsoft hates the word PayPal - mention paypay in your email and it gets killed, take out that one word and it goes straight through.
hotmail used to actively blackhole e-mail from smaller servers, no bounce, no warning, nothing. just accept it and throw it away.
i don't know if they still do that, but it was far easier to convince everyone i knew at the time that hotmail was garbage, rather than making any headway with their mail admins.
My only issue in the last 2 years was with Gmail, which I eventually discovered on accident was apparently because of my lack of appropriate ipv6 records in my SPF tag.
I didn't catch it for a while because so much of the email world simply assumes ipv4, but traffic happily exits my ipv6 interface when the destination supports it.
Even if it can't set the PTR it's self, it could at least detect if your PTR is missing or wrong, and display that information to you with some advice on how to resolve it and how important it is.
Additionally, you should test whether your PTR "smells" dynamic (i.e. contains part of your IP address in the PTR itself.) It may or may not be something within your control, but you should know as it may cause you problems later.
With dkim/SPF the lack of a reverse ptr is less important. I've had one email bounce as a result, even then the bounce included instructions to resolve the issue with their server.
From my experience, Gmail will send to spam everything from a small server that has no SPF (even when DKIM is on and OK -- which is strange because it basically has SPF built-in) and when the sender is not in the address book -- this is pretty enough for massive issues.
Well sure, but DKIM can sign the sender so it can do the SPF work connected with an envelope spoofing filter. Still, my idea was that a spammer who can put a valid DKIM signature won't likely have any problems with SPF.
I had heard of this too, and it was one of the reasons I was extremely cautious when starting to host my own email. To my surprise, I didn't run into a single problem like this.
I lack the expertise to elaborate much, but the gist is that people running their own server were caught by measures that were put in place to combat spam. I recall some providers (AOL?) used to be infamous for silently discarding your mails, which is very aggravating for the involved parties.
You had to jump through certain hoops to ensure that your mail arrived. DKIM/SPF are two such hoops (or related techs, anyway; both have detailed Wikipedia entries). Maybe it's not so much of an issue these days.
This happens as a matter of routine; the biggest problem with running your email server is you are not guaranteed that your messages will be delivered.
Please, please do not release this sort of thing for Ubuntu; target Debian instead. It's far easier to build Ubuntu support upon a Debian base than vice-versa.
Be aware: Rackspace (and surely many other cloud server providers) intentionally submit their IP blocks to DNS blacklists to dissuade spammers from using their hosts to send/relay mail. I'm not sure if the other big players do this but it wouldn't surprise me. Same goes for most consumer-class cable modem ISPs.
I wouldn't say that Rackspace intentionally submits their IP blocks for DNS blacklisting. Consider that on Rackspace's cloud you can build a cloud server and you will get a 'recycled' IP from their block. This IP could had been used by other people for other purposes. You never know who could've used your IP for what purposes. Having those blocks blacklisted makes sense when you think of that.
He's talking about something called the PBL (policy blocklist). The idea is it contains IP ranges that aren't "supposed" to send mail, like consumer ADSL ranges. People who want to send mail from home directly are supposed to do so via their ISPs SMTP servers, which may be configured to relay but only from IPs the ISP controls.
I am familiar with the SpamHaus PBL. It was my understanding that anyone could easily remove their own IP address from the list if required? Presumably that is what happens when Rackspace customers want to send legitimate mail. So why would it cause any problems for spammers?
One downside with ISP outbound relaying is SPF; if you want to use it you need to figure out how mail exits the ISP and keep the SPF records up to date; and even if you do, suddenly all the other customers of that ISP can fake mail from you while passing SPF.
Another downside is you lose logs of and insight in the mailq and the recipient smtp server responses.
One of my companies sends hundreds of thousands of emails a month from Rackspace servers. What can you tell me about this ? What blacklists, where ? We don't show up on any of the easily checkable ones.
We are of course CANSPAM compliant, use good email headers, unsub etc. etc.
I don't know about Rackspace, but if your mailhost is on Amazon EC2, you can use something like authsmtp.com or dyn.com who will sell you smtp relay service for cheap. Easy to setup and use.
Most of Digital Ocean droplets have their IP blocked. I spent several hours last week configuring a mail server only to find out my email were being rejected from hotmail and godaddy.
This is typical, I have just done the whole process myself a few weeks ago from scratch (fresh CentOS install) and now this comes up.
However I do not regret it at all. I had Linux experience and it took me one day and a half work to get it all working very nicely.
I am happy with my config: Postifx, Dovecot, RoundCube, SpamAssassin, ClamAV
Server supports unlimited domains and user accounts - SSL is required for all connections, I only allow IMAP and I have configured it with two valid free StartSSL certificates:
One under mail.domain.com (for email clients)
One under webmail.domain.com (for the webmail)
And both of course do not show warnings and green padlock is always nice.
The thing that scared me the most was outgoing emails being dropped - however to date I have delivered fine to all main email provider - followed a few simple rules:
Ensure you have both SPF and TXT correct DNS records
Ensure you have IPv6 configured properly (Google was rejecting due to this)
Set your reverse DNS
Set your machine hostname etc. in postfix
You seem to have learned a lot in the process. Can you point to any good resources that helped you, specifically with security concerns and outgoing mail reliability? I've already got a mailserver running with a similar setup as you, but I haven't invested much effort into security or making sure mail providers accept my emails (Gmail sometimes marks them as spam and AOL rejects them all together).
Mail-in-a-Box uses 'nsd', which is a non-recursive DNS server. I don't think it can be used to relay DNS queries in a DDOS attack. But if you meant something else, please let me know and/or file a ticket (I don't think I'll be looking here for replies).
Even better is to host your personal email on a server at your home. SMTP will retry sending mail if your server goes down for a period, so uptime isn't critical.
I've been doing this for several months with no issues, and it's pleasant getting out of google's targeted profiles.
I wish more protocols were built with such robustness and decenteralization in mind. If you try to host a web site at home and it gains some attention among netizens, you might have a bad day. Not because your connection is dog slow, but because all those people will complain about the way you host.
The internet is too darn centeralized, people can't just put services out there.
I often feel the same about how centralized the 'net has become. Two issues that seem to make it worse:
1. ISPs providing asymmetric service (and it's been getting worse, it used to be 10:1 dl:ul, but when I upgraded speed last year it grew to 20:1).
2. Why do we have a plethora of centralized services, when ideally a smart firmware in your home router can handle pages like about.me or simple blogs.
Setting up your own mail server is a thorny problem. I've been setting up my (family's) own mail server, and despite having lots of experience, it has taken a month already. Even when encryption, authentication, spam and sieve filtering, IMAP and webmail work, there's still no SMTP backup or alternative server in case you accidentally end up being blacklisted somewhere. I'm not sure investing money and effort is really worth it for just a few people's mailboxes.
The whole process keeps getting more complicated as the internet grows more hostile and end-user requirements increase.
This project sounds interesting and much necessary, but the real problem to me is the economy of scale. If 10 Unix guys sit together and set up 2 servers for 20-30 personal mailboxes, the time and money might be worth it. Setting up all those services, filtering and redundancy by yourself for just yourself – and monitoring them continuously – is wasteful and painful, even if you like to tinker with Unix systems.
There used to be a few geek-oriented ISPs around that offered SMTP/DNS backup, spam filtering and similar services so you could off-load some effort, but I guess they've fallen prey to low-margin virtual server business.
It does greylisting by default. Greylisting is awesome at cutting down spam, but we recently noticed that it was preventing a major mail provider from getting through, among others, because it seems that this major provider doesn't use the same IP twice (at least sometimes).
If ignoring email coming from a major provider isn't an option (it isn't.), then what are your options?
You can whitelist from GMail while still greylisting from everyone else. Simply query DNS for their SPF records and use that for the whitelist, Google is good about keeping that up to date.
GMail sends from a farm, it's random where the exit ends up being.
I only have that problem from GMail, Hotmail and Yahoo don't seem to have this problem.
Yeah a lot of greylisting is naive and uses "sender", "recipient", and "IP" to discover repeats.
If you're going to implement greylisting yourself it makes more sent to hash the sender+recipient pair, allowing IP addresses of the senders to change - which is what will happen with many of the bigger mail-hosts, gmail, hotmail, etc.
This is neat. I've been running my own email forever and appreciate the difficulty involved.
I do have a question:
Does hosting your own email truly isolate you from government surveillance?
If a million people use this to host their own email on AWS, what's to keep the NSA from ordering Amazon to tap into your data?
You almost have to host it on your own hardware in the garage in order to add a significant layer of isolation. Even with that in place there are ways to access your communications through your ISP. Harder, not impossible.
> Does hosting your own email truly isolate you from government surveillance?
Not by itself, no; the government can still get to you by getting access to the machine you host it on, or the switch the mail flows through on its way in and out from the outside world. Hosting your own just reduces the number of third parties the government can try to shake down to get access to your data.
I guess my point is that unless you have the machine in your home nothing really changes: They shake down the hosting provider and they are in.
In other words, what's the difference between buying email service from GoDaddy and renting a VPS from them to host it yourself?
Not being critical at all, just trying to understand if there's something fundamentally different about Mail in a Box that makes it more government-secure than other approaches. The description seems to list privacy from government snooping as the primary motivator. Is it truly effective at that? If so, why and how?
As I understand things the very idea of being secure from snooping is an illusion so long as your email is on a machine that you don't physically control in your home or office. And, even then, the best you can do is encrypt all traffic to make it more difficult --not impossible-- to capture your data.
It also means that (assuming one has some understanding of computer security) you stand a chance (albeit small) of potentially catching them in the act of getting access to your data (for some types of surveillance).
Are you serious? I just spent all weekend building myself almost the exact same setup from scratch. It was a royal pain. I wish I'd seen this a week ago. Nice work.
Sounds like its a little more than running the setup script, as you need to choose a hostname and then setup Reverse DNS. That could make a docker instance harder to create if there's customization needed.
The irony of Docker IMHO so far is that it almost makes installation that needs configuring with user-specific values harder, unless they know how to build docker instances themselves.
Although I do run at the moment my own email server with postfix, dovecot and the rest, I'm thinking for some time to replace that with Kolab [0] - a similar package of "everything email hosting related in a box" solution with an extra advantage of supporting active sync which I find very valuable.
I used to run my own mail server. I stopped because of two reasons:
1) You have to be your own 24-hour support, or accept that you will lose or bounce mail. You can be stuck with realising that it's failed while you're on holiday.
2) You have to be your own antispam. I've had my own email domain for fifteen years; it's on basically every spam list out there. Gmail is very good at this. At the moment I rely on my hosting provider who filters out most but not all of it.
3) You can have tedious issues delivering outbound mail, even if you're not using SPF.
In my opinion the right level to run email is "small cooperative" (10-100 users), where cooperatives share software, experience and maybe even staff but are administratively and physically separate.
This is cool, but it doesn't actually help you with mass surveillance unless the other side of your conversation is doing that too. What point running your own email server if your friend is on hotmail?
Also, even if you both run your own MTAs STARTTLS does not help much except to reduce the scope of the passive-only surveillance dragnet, because the configuration does not require signed certificates.
Finally, if both parties are technical enough to run their own email servers, um, why not just use GPG?
From a technical point of view, I put this squarely in "fun, but a boondoggle" territory.
> Also, even if you both run your own MTAs STARTTLS does not help much except to reduce the scope of the passive-only surveillance dragnet, because the configuration does not require signed certificates.
Exim and Postfix (and hopefully others to come) are both working on adding DNSSEC and DANE support right now. Which means in the not too distant future it will be possible to publish cryptographically signed data in the DNS that your MX has TLS support, and what the fingerprints are.
> Finally, if both parties are technical enough to run their own email servers, um, why not just use GPG?
OpenPGP encrypted email leaks all sorts of information/meta data.
The metadata problem is so hard that if you actually solve it you are not using email anymore. This is why I'm saying it's a boondoggle. Even in the best case where my mother and I are running our own STARTTLS secured MTAs, oh wait, timing and message size.
GPG gives you actual privacy but minimal resistance against traffic analysis. All this other stuff gives you NO end to end privacy, AND minimal resistance against traffic analysis.
Messaging is a weird special case and GPG is behind the times (because people are coming to realise that repudiability and FS are important), but it's still a heck of a lot better than even working TLS slapped on SMTP (which doesn't even currently exist).
The security of email will never be perfect; that is obvious. But there is still legitimate and real value gained from making small incremental improvements to it. If you can hide the To/From/Subject of a message from a passive or even active MITM (because of DANE), then that's a good thing.
In this situation, you are the one who gets the national security letter or warrant. Also, it requires a warrant because it's on your personal property. They don't need a warrant to search your ISP.
I think we're talking about different orders of magnitude of weirdo here :)
At this point the only people I know who could or would use Pond are people who can compose crypto primitives on their own anyway.
Also if we are going to go all Pond and do messaging from the ground up, we need to acknowledge that the global passive adversary is real.
EDIT: after replying to rickisen below, I'm starting to see your point. Maybe we need Pond or bitmessage or a resurrected mixmaster if real people think TLS solves traffic analysis.
Where are you hosting your email. Are you keeping it on say an AWS instance? OK flag that, all the plaintext would be there, and also, duh, it's still obvious that it's your communication because you're connecting to it, and maybe even paying for it.
OK, you host it at home on your DSL line? OK, so plaintext is harder, but oh wait. Yeah, your IP.
TOR hidden service or something? Good luck getting SMTP to work over tor.
So tell me again how TLS is going to help with metadata.
> It is a mail server in a box aimed to be deployed securely into any cloud infrastructure. It provides no user interface to send or check one’s mail but implements all of the underlying protocols that other applications (mail clients), such as Google K-9 for mobile devices, Mailpile, and Mozilla Thunderbird, can interoperate with.
The combination of Mail-in-a-box and Mailpile seems pretty exciting.
Since running your own mail server is opening a new vector for attack, I'd be curious how Mail-in-a-box is going to handle security updates.
Rather interesting. I still like building the whole thing with postfix + dovecot by hand myself. Perhaps I'm wrong in the head.
One thing to watch out for: Yahoo is a complete bastard to deliver to. One day they'll decide to block you (if they don't immediately) and the only way out is to fill in a web form which either allows you 6 months of delivery if you're lucky or blocks you with no hope of resolution for 6 months.
Oh wow. This sounds awesome! About a year and a half ago I tried to set up my own mail server on a cheap VPS, but could not get it to work for the life of me. It's sort of a stressful thing to set up as well - failure is both high-consequence and invisible, and I'm never quite sure if what I'm doing is secure or correct. Count me as a fan of the idea - I'll try setting this up later!
If you need email, but also web/ftp/git/mysql/dns, try Virtualmin - http://www.webmin.com/virtualmin.html - you can easily run this on a $5 digital ocean server.
You basically follow a web installation wizard, create a domain and then you can use Roundcube (included) or any IMAP client. Couldn't be easier.
A +1 for Virtualmin. Does all that th OP's server does (I think) plus can do it for multiple domains and has a simple web ui. Free, and their support forum is helpful.
Why has nobody just made a Docker image that anyone can download and use for free? Or even just a Dockerfile. That way it's a text file and people can contribute and improve the setup.
After toying with the idea for a while, I've settled on using a service like Mailgun or Sendgrid is the best decision when it comes to setting up email. Otherwise, it's just too much work for little in gain (as far as I'm aware). I'm curious to hear what others think.
I love Mailgun and use it for every project that needs to send out email, but you have to admit it doesn't address the OP's purpose: To decentralize and own your own email
I'm going to have to try this out - I tried setting up a mail server once and failed because it was too difficult. Now I'm paying $30/year for service from NameCheap.
I set up my own by hand and it worked great for about a year. Then emails started to not get sent and incoming ones were vanishing too. Then it all went to hell and just completely stopped working.
I gave up and went to fastmail. Setting it all up wasn't a problem. Maintaining it was a nightmare.
Huge advantage to hosting your own mail due to third party doctrine, so this is awesome, even if it is hard. Mail might be the one service I would be most reluctant to outsource.
Does anyone have a cheap VPS provider to recommend for this? I usually use digital ocean but from what i hear, most of their ip space is blocked everywhere due to spam issues.
I for one, am still sticking with qMail, qMail-toaster to be precise. Over the years it is proven to be very reliable even when it takes beatings like no other.
This is nice if your main goal is to take privacy back. For me, Gmail is not only free, but provides great mobile apps and email innovation like filtering, search, and a tabbed inbox.
In short, Gmail provides more value to me than being a simple SMTP server, so I'm willing to pay with my advertising eyes.
They're such a pain to use that since becoming the sysadmin in charge of our work email servers, I gave up running personal ones as well and just pay FastMail to deal with it for me - ain't nobody got time for that.
I often dream of taking a sabbatical and writing modern, user-friendly SMPT and IMAP[0] servers. If only so I could use them at work on my return and save myself a lot of time and stress.
[0] Possibly even with Exchange ActiveSync support - Microsoft licenses the protocol, but I've no idea what it costs http://www.microsoft.com/en-us/legal/intellectualproperty/IP...