One of the interesting things about this is the work on DMTP/DMAP, which are "Dark Mail" protocols that have security built in from the ground up (developed by Zimmerman of PGP fame, some Silent Circle folks, and Ladar Levison of Lavabit). It's a pretty neat thing that I hope sees widespread adoption. Fixing email has proven very difficult, due to the incredible inertia of existing systems...but, at some point we really do need to move forward on secure email communication.
I see references to ClamAV, and references to soon coming official docker images. So is this a complete MTA and delivery (IMAP/POP/etc) servers? I was digging around on github and the about page, but there doesn't seem to be info on how it does spam detection, or DKIM. Does it support signing messages?
I wish they had a more detailed about page and feature list, but I am still excited about this.
The ClamAV dependency concerns me; messages can only be passed to clamd via a plaintext stream. This seems to be at odds with the end-to-end encryption.
The ClamAV library is built directly into magma so plain text messages can be scanned without leaving the magma process space. I'd like to eventually have magma fork and scan messages in an isolated process space that doesn't have access to anything else, but that's a long term goal.
To clarify. We do not use clamd, or transfer the messages to another daemon.
There's a difference-- point-to-point encryption is what you're talking about. Anytime a message is transferred, it's done via TLS, but you can't be guaranteed that at any one point the message is kept encrypted. You can have this today -- just about any MTA has a configuration to only allow TLS connections, and then you can set up an IMAP server to only allow encrypted connections.
End-to-end encryption is the system whereby the moment the message leaves the sender's hands (phone, desktop, whatever) it's encrypted until it gets into the receiver's hands. This is your OpenPGP/etc. You could think of that as "Signal for email". This is apparently not what Magma is doing.
You can see that Magma is writing the email message to a file and instructing ClamAV to scan the file. This is actually worse than writing to a clamd socket, since (imho) it would be easier for someone to monitor a directory in a filesystem on a compromised Magma server.
Compare this to how SpamAssassin communicates with ClamAV-- it streams to either the port or socket:
Your slightly wrong. Magma uses mmap'ed temp files to hold the messages because ClamAV requires a file handle, or a file name. In theory the message only ever hits disk if memory is exhausted. And at least in theory, is never linked to the file system tree where it can be intercepted. If you know a better way, please submit a pull request.
If the MTA supports regular SMTP, POP and IMAP, then it cannot be said to perform end-to-end encryption. When a client accesses mail through any of these legacy protocols, the server must decrypt all messages before pushing them down the last hop. This used to be Lavabit's way of offering "encrypted" email.
If the MTA also supports some other protocol that affords end-to-end encryption, that's great, but now you need to use special client software that is either proprietary or seriously lacking in features. This is ProtonMail's way of offering encrypted email.
ProtonMail is currently working on a proxy called ProtonMail Bridge that runs on the client and talks to legacy MUAs. All communication between Bridge and the outside world is still encrypted. That sounds like a good idea to me, since I have no intention to stop using Thunderbird anytime soon. It's also probably the only way to bring end-to-end encryption to a large number of people, since it embraces and accommodates the inertia around email instead of arrogantly telling people to switch and wondering why they don't. When it comes to email, backward compatibility is everything.
I'm not sure if Lavavit's Magma can fill the same need; the documentation is rather sparse. But I would gladly run a local MTA if it means I can have my cake (end-to-end encryption) and eat it too (compatibility with legacy MUAs). DIME also looks like a much more open protocol than the vendor-specific stuff that ProtonMail is using.
While you lose a lot of the benefits of IMAP if the server does not have access to the content of messages, this is absolutely not necessary for POP and the only information which must be transferred in the clear to use SMTP is the TO address (and even then, only in the envelope, not in the actual message). Claiming that these protocols inherently are incompatible with transferring encrypted content is like claiming that any protocol that sits on top of TCP or HTTP is incapable of end-to-end encryption. What I think you are actually trying to say is "compatibility with existing clients without any additional software or plugins", but that is unrelated to the usage of SMTP, POP, and IMAP.
You're technically correct, but I don't think it's realistic to separate the payload format (plain unencrypted RFC 2822) from the various transfer protocols that have grown with and around it.
You mention HTTP, but do you think HTTPS have been as ubiquitous as it is now if people had to install a browser plugin in order to enable it? That sounds suspiciously similar to what South Korea has been trying with their infamous ActiveX plugins for their own national PKI. Nowadays they've moved away from ActiveX to actual browser plugins for Chrome/Firefox/etc, but nobody likes it as long as they have to install even a single plugin. "Compatibility with existing clients" really just means "compatibility with existing clients without any plugins". People don't like installing plugins.
Maybe a few years from now, every well-known email client from Outlook to Thunderbird to SquirrelMail will support an E2E encrypted email protocol (either a modified version of SMTP/POP/IMAP or a brand-new protocol) out of the box. Then, and only then, I think it might have a chance to become as successful as STARTTLS has been.
Please don't call ProtonMail "end-to-end encrypted email". They are mailing links to content hosted on their own servers.
The worst part is that they could easily allow interop and implement proper features in the client. Too bad they don't care and the competitor(s) (is there anyone else than Tutanota with a full e2e impl?) don't care about the users either.
I worked for a competitor a while ago, was ruined by a relatively incompetent CEO. The team is still fiddling with various small projects related to the field, but creating a proper solution without funding is quite hard, as it's a very extensive, challenging project.
You should look at the DIME spec: darkmail.info/spec specifically where it describes account modes. What we have working now is trustful mode, where the server acts a proxy to allow legacy clients to continue working. We're building DMAP clients which will allow cautious and paranoid mode, but it's taking awhile to get things built properly.
It's a monolithic mail daemon that speaks SMTP, POP, IMAP and HTTP, and uses DIME to encrypt messages when they arrive unprotected. It has preliminary support for DMTP and DMAP. Basically it's a replacement for your MTA and access server, which you'd run if you wanted a mail server that protected messages using an individuals password.
At first glance it seems like this will still expose who is emailing who to the network (and potentially other metadata), as well as relying on dns to function. Is that correct?
I guess I'll be the one to ask: Why write an email server with security as priority #1 in C? I'd understand if it was built off of an existing daemon, but it seems that this was written from scratch. There are many other options out there that are reasonably fast and don't have the pitfalls of manual memory management and undefined behavior that C has.
The most "interesting" thing is that they decided to effectively combine all of the components of a mail server into a single binary. This is actually advertised as a feature ("packaging is hard"), which is quite worrying because having everything in one address space means a minor bug in one piece now affects the integrity of all of the server components.
One reason for this could be that for an open-source application with focus on security it's very important to have many collaboraters, who look at your code and contribute. C is still very popular and many people know it.
The overwhelming majority of "high-level language" developers are, at an absolute minimum, familiar enough with C, or C-like languages, to be able to read/contribute to a C project.
Heck, most of us, yourself included, started off with C... it's just how it was/is taught.
Perhaps your C skills get rusty without use, or perhaps you're not a bit-twiddling magician, but you can still read it, and therefore think about it, and therefore write it.
It doesn't limit the pool of contributors... quite the opposite.
Now, if you want to have a debate about a better choice of language for other reasons, that might be fair game... but to imply there's a shortage of C developers is just absurd.
Most of the Python and Ruby developers I know are not comfortable writing C code.
More importantly, there's nothing in the world more dangerous than someone who knows just enough to get something working in C.
C is a harder language to learn than Ruby, Python, or Javascript. Unlike any of those languages, to write in C you have to constantly keep object life cycles, memory hierarchy, and memory layout in mind. Most of the standard libraries of Python or Ruby --- even simple string manipulation --- are idiomatically written and rewritten by hand in C. Even experienced C programmers forget, routinely and to calamitous affect, how machine integers work.
Writing something in C reduces the number of programmers who can contribute to a project.
And that reduction is going to get worse and worse over time. When I started my career, if you were going to ship code to clients, C was the industry standard way to accomplish that. A giant chunk of all working programmers used C as their daily language. Today, only a small minority of programmers will ever once write a production C module.
> The overwhelming majority of "high-level language" developers are, at an absolute minimum, familiar enough with C, or C-like languages, to be able to read/contribute to a C project.
This is completely untrue. My experience with Rust is that many, and likely most, Rust developers came to the language as their first systems programming language. That's one of the main reasons you see so many questions about the borrow checker: it's codifying existing best practices in C and C++, but most developers coming to the language aren't aware of those practices in the first place.
Your concern of memory management and undefined behavior is not unfounded. But in my opinion all languages have these problems, in the C the blame is shifted to the software engineer instead of the language authors.
Well, email is not that complicated, three letter agencies in the US have a history of hiding backdoors in technology, and one of the authors had his company destroyed by one of those agencies. If you're writing software for the tinfoil hat crowd, and you're using high level languages and existing libraries to do very basic things, you raise suspicion even if it isn't immediately justifiable... the suspicion that most major routers were routinely pwned wasn't justifiable in April, either.
Speed is almost never the primary concern with email, which is a world full of rate limits, throttles, spam filters, transient failures, tarpitting, blacklisting, inbox placement delays, etc.
>three letter agencies in the US have a history of hiding backdoors in technology
So the solution is to use the simplest tech stack possible and put the vulnerabilities in there yourself? If you're using this software, you have to trust somebody. I think most people would sooner trust that Java isn't backdoored than trust that this new C project doesn't have any buffer overflows or memory leaks.
I disagree. The more layers of abstraction you have, the more places there are for a backdoor to hide. Also, unless you're using OpenJDK, Java means downloading a binary blob which could be hiding any number of surprises, intentional, or accidental.
With magma I've taken the approach of trying to limit my dependencies to the kernel, and libc. Anything else I use is bundled and thus gets tested extensively for leaks, overflows, etc. That doesn't mean bugs don't exist, but it does mean if they exist, then the source is there for you to inspect and fix.
Different strokes for different folks. I think intentional bugs are far worse than unintended, and scanning a c project for buffer overflows is relatively trivial compared to scanning tons of libraries for flaws so obfuscated they made it through review.
In my mind technical analysis is incomplete at finding 100% of intended or unintended flaws in either system, but the DIY approach allows for trust in the authors whereas a deep stack makes attribution muddy.
>scanning a c project for buffer overflows is relatively trivial compared to scanning tons of libraries for flaws so obfuscated they made it through review
You would have to scan the C project for buffer overflows and memory leaks in addition to scanning the tons of C libraries that it uses[0] for buffer overflows and memory leaks (note that openssl is in that list, among other massive libs). This is not even taking logic errors into account. There is simply too much to consider all at once, and being in C just makes it that much harder have a reasonable sense of security.
You're also implying that it's easier to spot intentionally backdoored C than intentionally backdoored Python, Go, Java, etc, but I have no reason to believe that that's true. Furthermore, the number of eyes that have been on those projects is far higher than the number of eyes that will ever grace Magma.
As far as I understand it (which is quite limited, I haven't been able to find a white paper or whatever about DMAP thus far), there's a fundamental difference in what layer of the stack security resides in.
JMAP is a vaguely IMAP compatible protocol, in that the mail storage is probably just Maildirs or whatever (with some indexes to make it fast). It is not a complete re-design of how email works. Encryption is just in the network layer between client and server; the server, if compromised, would compromise the mail (unless other encryption features are layered on top, as in the current email standards). That's a perfectly reasonable design decision, given how email has always worked, and the problems inherent in pushing encryption to the client...particularly if the client is a web browser.
DMAP is not merely "IMAP with JSON over HTTP" (JMAP isn't just that, but that's a reasonable enough short description). It's a fundamental redesign of a mail client protocol. Again, I haven't read the formal spec or anything, and there is no source for DMAP yet that I can find in the repo (DMTP is there, but no DMAP), but given what it's promising, it can't possibly just be layered on top of IMAP without significant changes. DMAP seems like a very long-term project; maybe years away. I don't know if there are any implementations, yet.
And, JMAP is so new as a public thing that I would guess it wasn't on Levison's radar until recently. It recently began the formal standards process, which is cool. But, it's also not really got any complete implementation (just a PoC proxy and an incomplete implementation in Cyrus with plans for Dovecot to get support at some point, too).
So, I think it boils down to a few things: JMAP doesn't solve the problem DMAP sets out to solve (which is end-to-end encryption with at-rest encryption on both ends), JMAP is also very new and has no production-ready implementations on client or server, and these are both open sourcing of projects developed as commercial tools (JMAP comes out of Fastmail development, DMAP comes out of Lavabit and Silent Circle).
Also, this does appear to have an HTTP server...so, probably something quite similar to JMAP, though certainly not talking the same protocol (since it pre-dates the publication of JMAP details). It could probably be made to support JMAP without a huge amount of work, since it already supports IMAP and HTTP.
If so, they haven't Open Sourced the implementation they're using (they have open sourced a JMAP-IMAP proxy that is pretty nice, but I probably wouldn't want to use a proxy in production). I don't recall what they've said about how they're using it, and if they're using it or some precursor to JMAP on their production systems.
JMAP isn’t all standardised and set in concrete yet (there’s an IETF working group for it [1]), so we’re not enabling JMAP for the general public just yet, nor is our branch of the FastMail web UI which uses JMAP deployed.
DMAP is designed to work optimally with encrypted message objects. I looked into making it a JSON protocol but JMAP didn't adapt well and was even more complicated than IMAP (if you can believe it).
Magma is an email server, supporting many of the common email protocols, designed with security in mind. It was designed and used by the guys at Lavabit (of Edward Snowden fame).
It combines outgoing (SMTP) and incoming (POP, *MAP) functionality, so it would be like using Dovecot and Postfix (rival mail server products) together.
It requires that you have a relational database (the documents describe MySql) and a key-value store (the documents describe Memcache) to support it and the documentation suggests it's designed for Linux (specifically with instructions for CentOS, which is a Red Hat distro). A mail server is generally installed on an "always-on" computer connected to the internet-connected network.
Note that even though it's "designed with security in mind", I don't see any mention of a security audit on either the linked page or the GitHub repo.
I don't mean to be a pain, but before anyone can try to answer that for you, you need to define 'reasonable' and 'secure' in terms of your environment and needs.
Are you Just Some Person(tm) with some technical chops, who otherwise isn't going to attract a lot of unwanted attention?
Are you someone who does things that might annoy others enough to make them want to mess with you? Who are they and how annoyed might they become?
Are you an aspiring Ed Snowden?
Do you have confidence in your skill set to 'know' that you've configured this and everything else on the machine securely? If so, do you watch the disclosure lists for vulnerabilities, enforce configuration, audit and monitor, and generally stay on top of these things?
Etc. Again, I'm really not trying to be overly pedantic, but there's just no such thing as 'secure' without qualifying it with a description of 'against what'. Similarly, what's 'reasonable' for me - someone who's worked with this stuff my entire career - is absolutely bonkers to suggest to a non-technical person.
Anyway, all that said, this doesn't look ready for prime time to me. I'm certainly interested in it, and might play some when I get a chance, but even after it looks ready for use, I'd likely wait for a while to see what other people find before deciding to use it in production. My mail system runs Postfix and Courier with various support systems. Mortals can indeed run this sort of setup, if they're willing to learn a bit. It isn't rocket surgery, but it takes a bit of work.
I don't know about the parent comment's requirements, but I'm:
1. Someone who has some servers and a little tiny bit of technical knowledge. Or, to be exactly, I happen to already run Postfix+Dovecot+rspamd for my email.
2. Someone who is interested in personal email security/privacy, in particular in terms of both in-transit and on-storage encryption. I can't realistically watch for every CVE and patch all the holes on the day 1, so I would always love something that would let me trust my servers and network less.
I don't have any specific threat model or a designated adversary. I just see "hey, there's a some new stuff that looks promising" (this submission) and wonder if it could improve my life. However, the documentation is sparse, or, I'd better say, nearly non-existent (or I'm looking at a wrong place). So, asking if someone had already spent time to evaluate or had actually used that and can share their experience is a logical choice.
Thus, I just wonder - is Magma for me, can I run a secure personal email server with it? Like, use it to replace my Postfix+Dovecot system? Can I expect it to be no less reliable than my current setup (which is just your average run-off-the-mill configuration, documented in every other "how to set up your own mail server" article)? Is it correct to expect it to add some value in terms of security? What the exact those improvements would be?
As I (very briefly) mentioned, it looks to me like Magma isn't ready for prime-time yet. I've seen reports that, for instance, the LDAP server allows completely insecure configurations (cleartext passwords on the wire) without so much as a warning. That sort of thing should not be possible with a product like this, and makes me think they're gluing together pre-existing code, which is also a little worrisome[1].
Beyond that, I tend to be pretty conservative with things like email. What you currently run, assuming it is configured correctly, is solid; I'd personally be hesitant to replace it until other folks have bled over the inevitable first bugs and it has been in use long enough for people to learn how to operate it, figure out any weirdness, etc. At the least, you'll be skipping what will probably be several point releases right after whenever they actually call it done.
TLDR; if I were you, I'd hold off. (And am personally holding off.) I'm very interested in this, in theory, but even though I am technical, I'm not a cryptographer or a security researcher, and even if I were I don't have the time to audit it. I want those folks to beat it up before I wade in.
[1] Nothing wrong with that in principle, but combined with the jagged edges, it makes me wonder how much attention any grafted code is actually getting.
mm_set is being used. As I recall that pattern was inspired by looking at a libc implementation for memset. The combination of using volatile and the asm("") statements keep the compiler from optimizing away what it thinks are unnecessary memset (like a wipe operation). It's possible that solution doesn't work on every system though. So, how would you improve it?
Basically just the right inline asm. But the project as a whole is a bit of a mess, multiple 'memssets' will not do anything to make things more secure than just one.
Is it used anywhere? Whenever I find an obvious bug in a large codebase, it's usually in a section that turns out not to be used. Or used in a section that isn't itself used anywhere, etc.
So, this is really cool stuff.