Sadly, Google and Microsoft are in the business of getting their users to use their own gmail and outlook clients. They don't want to be invisible IMAP/JMAP back-ends, they want to own your e-mail experience.
MSGraph is slower and not as convenient for email compared to JMAP. In addition to requiring more API calls for the same result, MSGraph starts throttling with HTTP 429 responses after several calls, making the initial synchronization much slower than JMAP or even IMAP.
Hopefully, next year there will be JMAP support for Contacts. As RFC drafts, there are already JMAP specifications for Files and Calendar...
No doubt JMAP is superior for the problem it solves. I wonder if it will matter in the end though. I have the impression MS is positioning it self to be the only viable AI platform where mail and calendars are just a tiny part of that. So while not really competing in the same domain it may be that both IMAP and JMAP may feel as relevant to the world as IRC or XMPP in the not too distant future.
Sure, and we're the vocal (?) minority here. The SOHO/corporate/family world will uncritically use whatever shit MS regurgitates for them for all the usual reasons.
It does, but IMAP is clearly not a priority, and they certainly aren't interested in JMAP. They support IMAP (with caveats) because you can't really offer e-mail without IMAP, but they want you to use their GMail app.
Normally, email providers let you organize into folders. And IMAP (if I understand correctly) DOES NOT allow clients to see those folders and replicate the folder structure.
But Gmail doesn't have "folders". They have "labels". I wont go into all the details about how Gmail labels work that frustrate and annoy me, but the point is that IMAP clients cannot see labels, they can't replicate it on the client end, etc.
This caused a legitimate problem at work when the finance dept. wants to use this service that reads emails to automatically scan invoices into their system and whatnot. They wanted to be able to only have a subfolder automatically imported into the program, because too many of the emails were not relevant to get imported, creating more work to clear them out.
We were not able to achieve a solution with a simple folder for the users to move emails into to get them imported. Instead, we had to set up a second email account specifically for this importing service, and whenever finance wants something imported, they forward the email to that second account.
The problem is that in IMAP the concept of labels doesn't exist. Thus they are implemented using folders, which is a concept in IMAP. But that issue is, a message is then in multiple folders (to reflect that it has multiple labels), but in the way this happens in IMAP, they are different messages, even if the content is the same. And thus I can imagine, automated programs, not aware of this, may for example see 2 invoices (because the email has 2 labels) rather than 1.
IMAP has a concept of flags/keywords (https://datatracker.ietf.org/doc/html/rfc9051 2.3.2) and there's a whole bunch of servers and clients that support using it in a similar way as labels - Thunderbird, Zimbra, dovecot, etc.
It's not a feature originally built for labelling like Gmail, so it has a few complexities ("Servers MAY permit the client to define new keywords") and enough clients (especially mobile) that don't support it - but they could just like Thunderbird or some webmail clients.
I've used it basically exactly like Gmail labels for years on TB on desktop.
> Normally, email providers let you organize into folders. And IMAP (if I understand correctly) DOES NOT allow clients to see those folders and replicate the folder structure.
JMAP uses fewer resources server-side, and with their scale, it will probably reduce operating costs. But it will depends on how much they will save compared with devel/migration costs.
It may use less sockets but that doesn't necessarily mean less resources. Parsing JSON is expensive, especially when there are binary types interleaved in it.
Can you give an example of expensive binary data parsing?
Having implemented JMAP and IMAP protocols myself, I haven’t encountered a need in either protocol to send binary data to the server for sync/search operations that would require the server to perform expensive parsing.
JMAP offers many improvements, such as a stable messageId for each message and a state mechanism that allows the server to be queried for changes since the last saved state. This avoids the need for numerous IMAP SELECT commands per folder to check the state using CONSTORE/QRESYNC.
If CONSTORE/QRESYNC aren’t supported by the client, it results in very costly chunked queries just to verify if message flags are still the same.
The same applies to SEARCH—if a user has many folders, it requires multiple network hops to SELECT and query each folder. With JMAP, this can be done in a single API call.
Nah, JMAP is optimized to make fewer and cheaper calls to the server. I agree that it was a mistake to use JSON for JMAP but I doubt the overhead of using JSON matters in practice compare to the benefits. Parsibg JSON with SIMD instructions is very cheap.
I'm not sure it does if I'm honest. The encapsulation via JSON is quite bad and transactional model of HTTP is non-existent. One of our key problems on our platform is deciding on how to handle failures like that in transactional systems. Email is to some degree transactional due to the distributed state. It's going to get ugly.
The big benefit with JMAP is the state strings and /changes operation which allows even in the face of failures to recovery quite cleanly. It's really more a data synchronising protocol (RFC8620) with email support on top (RFC8621) and the new protocols coming in.
Section 3.10 of RFC8620 scares me. I know a lot about that and a lot of the complexity around that trite point is missing.
Note I've designed and built a complex messaging system that runs over HTTP over unreliable connections and run it in production for 15 years so I know what can and does go wrong.
Have you looked at their list of what doesn't work offline? It seems like it's mainly attachment management limitations than limitations of managing the emails themselves.
I have no idea whether JMAP is any good, but I find all the hype utterly baffling. JMAP is supposedly awesome. There is real thoughtful engineering going into offline support: https://www.fastmail.com/blog/offline-mail-storage/
And yet: at least as of a couple months ago, there isn’t actually any offline support! The Fastmail first party app does not work at all offline. Complete failure. Does not even try. Oh, and it has dramatically worse threading support than even Gmail or Outlook.
I don’t get it. POP3 is awful and kind of works offline. IMAP is old, clunky, and works fairly well offline. JMAP is supposedly the new hotness, but the paid first party experience will not even try to load offline. This is table stakes! Eudora could do this. Every version of Outlook ever could do this. Thunderbird can, and always could, do this. The usual command-line clients work offline if configured appropriately. Heck, Google went above and beyond and made Gmail’s webmail work quite well offline if you care to set it up, and I think they did a bunch of early work on service workers to make this possible. Fastmail, please stop pitching your fancy protocols until you can get your clients up to the state of the art as of twenty to thirty years ago.
edit: Huh, mobile offline support is in beta as of December 16. No way!
Similarly, I wish I could find some iOS mail app that allows me to have the have the full mailbox locally. I use Gmail with the Mail app on iOS, and tried multiple mail apps to try to find one that will download the whole mailbox with all attachments and let me search it locally quickly, but seems that doesn’t exist.
On Mac I’m using Mailmate and liking it.
This is just a step toward offline support. It's not available yet, because Android comes first.
The app is ready for offline use, but I don't want to implement it without 'full text search.'
I plan to use SQLite FTS5.
My initial tests with search show that offline search is storage-expensive, which could be problematic for mobile devices.
So, the edge cases have started to appear. But it's the most requested feature, and the most interesting/challenging to me :)
Yeah for high-end phones with lots of storage, fully offline full text search would be really nice.
Often I know that some email with a substring exists and I search for it in Mail app, and it goes off to do a server search… And when you need it most, you probably have 1 bar of range, or have no cell service and no available WiFi. Example: pulling up reservation details while traveling in a foreign country.
Or even worse, sometimes I look at some email in Mail and it’s there, and then hours later, while on bad network conditions, I look at it again and see “This email has not been fetched from the server. Download?”, which then just hangs.
I have a 1TB iPhone and a 2TB iPad. My main Gmail account is not large by any means - ~50k emails and 5GB space. Even though I have all this space on my iPhone/ipad available and willing to commit a bunch of it to email with all attachments and full text index, there seems to be no way to do it.
The Gmail app has a slider for “days of mail to sync”, but the max is 90 days, and not “All time” like one would expect. Anyways, good luck with Mailtemi!
Unrelated to the discussion of JMAP, but I had the pleasure of helping host the mentioned Inbox Love conference, assisting Josh Baer and Jared Goralnick. That conference was such a fun one!
Targeted, focused conferences like Inbox Love, with 150ish or fewer attendees, are by far my favorite because you can actually get to know folks, ideas flow more easily, and everyone is focused on approximately the same thing. Much better than huge, multi-track conferences. We should host more of those as an industry.
On the flip side, I would like to see a free of cost email provider offer JMAP as well. It could be a limited version in the sense that something like the free version only has limited storage or something but it offers the full JMAP experience ^TM or at least that's my expectation to use JMAP.
How come there are no free of cost email providers that support JMAP?
You can use https://stalw.art. Compared to other email stacks, it is very easy to set up. Its support is on par with Fastmail.
I develop a JMAP email app (https://mailtemi.com) and have run equal tests against stalw.art locally as well as Fastmail.
Neither Fastmail's JMAP for third-party apps nor Stalw.art supports Contacts/Calendars yet.
Fastmail said that it will be enabled once it passes IETF standardization.
Stalw.art also mentioned they will add support after the standard is finalized.
I know a couple of people who run large email stacks (500k+ users) and this isn’t on their radar and there is no interest in it. There is a little disdain as they are comp sci people and don’t like JSON at all (I share that concern).
Email is probably the most conservative, reluctant to change set of protocols there is.
I’m not exactly the biggest fan of JSON either but given the clusterfuck of established email technologies, moaning about JSON is a little hypocritical.
IMAP is both excellent and, annoyingly inconsistent so it's much more of a pain to develop reliable parsers for. I'm pretty happy to use IMAP generally, but I'm also... MODSEQ is wrapped with a () in some places, not but STATUS HIGHESTMODSEQ:
. SELECT INBOX.Archive
* 1239 EXISTS
* 0 RECENT
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen $X-ME-Annot-2 $HasAttachment $IsNotification $IsMailingList $NotJunk $CanUnsubscribe)
* OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen $X-ME-Annot-2 $HasAttachment $IsNotification $IsMailingList $NotJunk $CanUnsubscribe \*)] Ok
* OK [UNSEEN 1221] Ok
* OK [UIDVALIDITY 1108730350] Ok
* OK [UIDNEXT 2231] Ok
* OK [HIGHESTMODSEQ 40306873] Ok
* OK [MAILBOXID (210306ee-5833-456b-bede-6d04757128b3)] Ok
* OK [URLMECH INTERNAL] Ok
* OK [ANNOTATIONS 65536] Ok
. OK [READ-WRITE] Completed
. FETCH 1 MODSEQ
* OK [HIGHESTMODSEQ 40306873] CONDSTORE enabled by FETCH MODSEQ
* 1 FETCH (MODSEQ (39570709))
. OK Completed (0.000 sec)
So it is just coincidence that there are zero good IMAP clients out there? I have tried a lot of clients and nobody has managed an implementstion which is works well in practice.
And it is a pretty bad query language compared to something like SQL.
Email is probably the most conservative, reluctant to change set of protocols there is.
And it'd better stay that way, as it's also the most widely used set of standards for communication across borders, that people rely on about as much if not more than traditional telcos now.
(You should know why the common-sense comments are downvoted --- this is basically an advertisement article, and they don't like it when their $shiny_new_thing is actually shown to be worse than the existing solutions. A huge clue is when the adjective "modern" is thrown around like it's a good thing.)
As one of their customers, I really appreciate that they don't upend everything all the time with pointless features.
Constantly reinventing things and piling on pointless features is one of the things I dislike most about Google, their chat solution is especially egregious for this. It only causes frustration and increases the time investment required to be proficient with the products.
They have added features though. They're just not invasive. They added masked emails, for example. And if I look at the settings right now, I notice they've added customization of what swipes do.
I too like the feeling that they're stable. I guess they're just tactful with their additions.
I switched a couple years ago because the masked emails and i was tired of histing anonaddy and i couldnt be happier myself. Easily the best mail provider i have used and i have used a bunch of them (including all the big guys, a provider using zimbra,protonmail, etc)
Nope, IMAP can't provide the kind of smooth reliable experience that the Fastmail mobile app provides. Now they obviously could have done it in many different ways, but I know from experience with IMAP that it can't do it.
The amount of times IMAP has resulted in duplicate emails when moving mail between folders for me is enough that I eventually gave up on attempting any kind of organization back when I was last using it.
Anyway, It has been some years since I last used IMAP. I had issues with Outlook, Thunderbird and Nylas N1 while connecting to Gmail, Exchange and Office365. It was a fairly small mailbox getting about 20k mail a year and keeping about half of that. Usually moving about 5k mail around at a time (end of year).
It got so bad that I had to get familiar enough with the IMAP protocol to make my own client program to handle deduplication overnight. Shortly after that I decided to just give up on trying to keep thinks organized.
I am in Australia and this was before the NBN, so obviously it was always on shitty connections.
I have considered writing an E-Mail client, so I’ve done a little research on the ecosystem.
JMAP seems like exactly the protocol I’d want to use to avoid much of the IMAP pain, but there’s almost no ecosystem.
If I wrote an E-Mail client for JMAP, I’d basically make it for Fastmail and for Fastmail only - and I couldn’t even do so without paying Fastmail. That’s untenable, I’d be 100x better off targeting the GMail API.
An IMO perfect way to jumpstart adoption would be an IMAP-to-JMAP proxy, which I don’t think currently exists (maybe due to basic deficiencies in IMAP, I’m not sure). It would allow people waiting to develop and use modern clients relatively easily.
Currently the best way to get non first party JMAP infrastructure seems to be to host Stalwart (https://stalw.art/) and do some trickery to forward your E-Mails there, which hasn’t been worth the effort to me so far.
I get that this is a huge effort and that it takes a long time for protocol adoption (especially if the big companies aren’t behind it). I don’t know why Fastmail has developed mobile apps but no desktop applications so far. One justification could be higher usage on mobile, but it still leaves desktop users out (of JMAP).
Without more email clients supporting it, mail providers don’t have any incentive to support JMAP. Mozilla Thunderbird started looking at JMAP but hasn’t progressed on that all these years.
> In 10 years time, I hope to post about how Cyrus and JMAP have taken over the world
I believe the biggest hurdles are other email clients and providers not adopting it. The biggest threat is Microsoft, as usual, pushing its own protocols and client and using FUD to brainwash CIOs into believing that any protocol outside its own is a major security threat that just cannot be handled.
If either Apple or Google could be convinced to implement and support JMAP, this could take off a lot faster.
If JMAP is to be successful, having support in Thunderbird is a must have first step. It's not helpful if the same party that provides the mail service provides the frontend as well. In that case you might as well leave out the open protocol and just push the browser-based client.
I like Fastmail and use it because it provides a service, and does so well (which includes support you can actually reach out to and talk to a human who will actually help you). I also like that my identity and daily e-mail experience is in no way hard-linked to them. I have my own domain, and I use Thunderbird and K9-Mail (now also Thunderbird).
I also like that I can log onto to their website and do things there if needed. This is mostly managing masked e-mail addresses and the occasional check on my subscription settings.
> If JMAP is to be successful, having support in Thunderbird is a must have first step. It's not helpful if the same party that provides the mail service provides the frontend as well.
And if you're going to fund development, also put it in Dovecot.
They complain about complexity, then add another two layers of complexity in their own protocol?
I think POP3 is the simplest standard, and have also written a basic IMAP client. Parsing IMAP isn't as easy as a binary protocol, but it's definitely not at the level of HTTP JSON bloat that seems to have infected all "modern" protocol designers. I can use POP3 reasonably easily from a netcat (and have done so many times in the past), and IMAP is a little harder but doable. I don't expect that to be doable for JMAP which is text-based like the other HTTP JSON bloatocols, but unlike the earlier text-based standards like SMTP POP3 IMAP IRC MSNP etc., it seems to have all the disadvantages of a text-based protocol but none of the advantages.
POP3 has no locking. It won't work with multiple clients, and that is awful. But I agree with your assessment. We need good utilities which make JSON human-readable, and which turn human-readable into JSON. Might as well not use JSON, except for internal traffic.
If only one client writes data or if you constantly do full sync (quite expensive on resources), yes. You can't even have different maps to sort content though. You don't follow states. POP3 just doesn't scale well in the end. If you're in 1995 and you get 3 emails a week and get really excited you got an email, then it works very well (I still hated the full syncs on dialup though). I used POP3 for several years in the 90s, even wrote my own client (a very simple protocol) Then end 90s, I used Pine and then Mutt directly in a shell on my ISP, with IMAP and Maildir; a relief! I've never used JMAP, since 1) I quit self-hosting email due to (justified, given spam) complexity of the stack and 2) I don't use Fastmail but Soverin.
It would be great that the two big email providers (Google and Microsoft) implemented and supported it.
It would make so easier and reliable to have a single client that works really well across personal and business email accounts, for example.