Hacker News new | past | comments | ask | show | jobs | submit login
Why CardDAV took so long (fastmail.com)
176 points by masnick on Dec 1, 2015 | hide | past | favorite | 67 comments



"Only this one user had problem. But he had pretty bad problems, and I wanted to fix this properly."

That's why I'm a Fastmail customer.


I submitted a bug report once and it was fixed the next day, I was invited to try it out on their beta version of the website and it was rolled out on all the servers a couple days later. These guys are truly providing a fantastic service.


Similar experience. Fix with inability to respond to some crazy github-generated email addresses rolled out publicly in less than 48 hours. It helps that I was able to tell them "I am pretty sure it's your regex on valid email addresses because I can reproduce it with other emails like $this"

I also recall they fixed an issue where "--" at the top of an email I was replying to was making the whole thing look like a signature and when I tried to send it was warning me about an empty message.


I'm going to repost an earlier question I had about FastMail ...

It seems like everyone I read loves using FastMail for personal email/calendar/contacts.

What I haven't found is many Businesses using FastMail for corporate email/calendar/contacts.

Can someone compare FastMail (for business usage) to Office 365 and/or Google Apps?

Original question, https://news.ycombinator.com/item?id=10511000


Office 365 at least has several well known bugs:

1. Silently dropping email (no smtp transaction bounce, not in junk, no rules in place) for a specific single sender at a domain, where email from other senders at the same domain is accepted. Hundreds of people have posted about this online with no fix or acknowledgement from Microsoft.

2. Overly strict hardfail SPF policy by default for new customers who have no idea of SPF and should be using softfail. These customers often use third party systems to send out hundreds of account statements. These get blocked by other email hosts. The customers themselves have no idea and never correct their SPF, saying something like "it must be an issue on your side".

3. Blocking entire /24 IP ranges including good senders with good reputation, on the basis of only a single spammer in the range. This particularly affects those in datacenters, even if you are on a dedicated machine and your IP is good.

4. It may be that Office 365 does not track or measure domain reputation at all, which makes things like SPF or DKIM of little positive use, and introduces loads of spam false positive bounces.


It also kicks me out of my account multiple times per day. Then, if I get kicked out overnight and try to login the next morning, the form key expires (CSRF I assume?) and the page just stops working. The only way to login at that point is to re-type the address in the browser.

This is for Outlook365 using a custom domain for my work.


Sounds pretty awful choice to me.


We tried using FastMail for our (small) company, but switched after ~8 months because we started running into deliverability issues and countless issues with shared calendars being unshared (or sometimes losing events).

We also couldn't use any of the nice sales tools that integrate with Google Apps (or any of the tools that make managing calendars easier, like https://claralabs.com/ or http://boomerangcalendar.com/). We're now on Google Apps and very happy.


I've seen a couple of nice tools integrated with Gmail and its web UI. The ones you mention are like that. However my problem with such tools are that they are not operating on standards. For one I wouldn't want to be locked into Gmail. But more importantly, tools depending on proprietary platforms usually have a short life. For example people loved Mailbox from Dropbox, while I was the one complaining that it only works with Gmail and iCloud. And I was right, Mailbox is dying.

And also these tools get access to all of your email, which is a privacy concern. And in the context of a company, that can mean industrial espionage. Personally I cannot understand how people can trust things like IFTTT, but whatever.


There can be odd behaviors with third party apps, too, and if you get dependent on that service you're at their mercy if they don't fix the issue.

I mentioned in a separate comment, Sunrise substituted a generic email address for mine in an invite response, but with my name ("Brer Lapn <invite@sunrise.am>"), which when someone replied to the group began to propagate across multiple co-workers' Outlook as the autofill address for my name. I missed email from some people for a week before one of my coders noticed it and let me know, at which point I had to get colleagues to forward me any message with that address, contact the sender and ask them to remove it from their Outlook autofill settings, and repeat ad nauseum until correspondence started coming to me again.

I sent a bug report, they responded "we'll look into it" and never heard another word about it, so for all I know it is still happening. If we'd have integrated that into the company processes rather than just my using it to consolidate my calendars it would have been a serious problem.


100% agree with you, especially on the privacy concerns. But you have to pick your battles and, when we were looking into these tools, our battle was not the privacy of our company email or calendar.


I guess it depends how much of your companies IP is sent around in internal emails...


Yeah, that's a good point. Nearly all of our work is open source, so we weren't too concerned about losing IP.


I'll have to concede the losing sharing. There are still some frustrating things there, the DAV ACL model doesn't map very well to the Cyrus model which is based on IMAP permissions and there were a couple of bad bugs in there until recently. We're still running patches that I'd like to remove, but at least we finally understand the cause of lost acls and fixed it.

I'd love to know more about deliverability issues - that's something we monitor closely, and we want to know if there are any silent blocks/drop rules anywhere that we can't monitor for!


Integration of third party tools with Google Apps is nice, but having been burned using Sunrise and having had more than enough troubles using Gmail outside of the web app (such as via Outlook or Thunderbird) or native Android client the shine on Google Apps is a bit dull for me at this point. I'd be much happier working with services that use well-defined open standards that work with my choice of client software.

I know that Sunrise isn't a Google-apps specific software, but after: * a co-worker sent me a screenshot of a meeting invite I had accepted via their app (along with a facetious question "You get a new assistant? Who's 'Sunrise' and is she cute?") that makes Sunrise branding too prominent for it to be appropriate for a professional environment, and * finding out that instead of masking a response with my email address it had sent it from a generic "invites@sunrise.am"-style email but showing up as my name "Brer Lapn <invites@sunrise.am>", and having to then contact over a dozen colleagues who had been emailing the black hole address at Sunrise--- I'd say these services are less suitable for business purposes than the standards that Fastmail uses (and publicizes). Fastmail has never screwed around with my emails or calendar invites, and unlike Gmail I can use multiple different clients without it filling my drafts folder with autosaves.

TL;DR, if I had a say I'd be running our company tools on Fastmail rather than Google Apps (although deliverability is definitely something I'd consider a showstopper, I've never had an issue with FM).


Fastmail is email and calendaring not for cloud based based document editing / file storage.


It's clear from tiffanyh's comment that the question is about the mail/calendar/address book part. Though, I cannot give an answer to that question. From my experience with a personal account, I can say that Fastmail permits much more customization.


They aren't independent questions.

Why would you get cloud based document editing and storage from microsoft or google, and then not use the freely included and well integrated calendaring and email that either provide.


Trust me - Microsoft's email and calendaring is anything but well integrated, it's a flaky, unreliable mess.


Can you please expound on how it's flaky and unreliable? I'm a user and am wondering what I'm missing. :)


We do have file storage via WebDAV or ftp, though it's not as full featured as companies that specialise in it. I wrote that up last year as well. It works OK, though it will be nicer one we spec out jmap file storage api.


I didn't even get into the remaining compatible with the existing contacts system. CalDAV was green field, but CardDAV had to integrate with the web, with spam scanning (whitelist/score bonus for known users), with mailing list address book groups, ...


The only feature I miss with your CardDAV implementation is contact photos. Everything else works very well so I leave email and calendaring to FastMail but I'm back to self-hosted contacts for now. Really hope this can be added soon.


They do work, but only if you upload them via CardDAV - we don't have any UI for them in the web client yet.


Interesting, I'll have to try again. When I first used the beta I imported a VCF exported from ownCloud (which I think stored the images as data URIs). I'll try to figure out how to upload them via CardDAV now, thanks.


To add something to what @robn already said - with Google Apps the resolution you get on contacts syncing is 96x96 pixels. Not sure what resolution FastMail supports, but seems to be OK on my 1440x2560 phone.


Miserable, this kind of stuff is why I don't run my own mail server any more. Happy Fastmail customer since 2006.


It's really hard to run you own mail server. Much harder than a web server. This is bad for freedom on the internet.

http://www.ebb.org/bkuhn/blog/2015/09/15/email.html


It is not that hard to run your own email server, what is hard is to get your mail into recipients mailboxes. The big players in this market have basically decided to block (at random) all mail that doesn't come out of a known email service.

I have just given up running my own mail server after more than 15 years because of this problem. Five years ago I had no problems, but these last 12 months have made me decide to give up and just pay.


It's true it is harder; definitely much harder if you're not a sysadmin by trade. These days to get mail to deliver you not only need to support SMTP, but also TLS, SPF, DKIM, DMARC at a minimum. You also need to make sure that you aren't inadvertently a source for relaying, or non delivery report (NDR) / delivery status notification (DSN) bounce source (vis a vis spam with faked return paths etc).

In short if you don't / won't / can't do the above you are indeed much better paying someone else to do it (preferably at scale with attention to detail like fastmail).


I used to do all these things (not that I am a sysadmin by trade, but after 15 years you get to know the job). Even doing all this the big email players (Microsoft in particular, but Yahoo and Google are also guilty) will just ditch your emails without rhyme or reason if it is not coming out of a known email service.

This didn’t used to happen, but it looks like they have now outsourced their spam filtering to the email service providers who now act as the gatekeepers to email ecosystem - email from a known service == good, email from business that has been around for more than a decade == spam.


The irony is that nowadays most spam I get is from @gmail or @yahoo addresses, so all they've accomplished is lock-in.


I have been running my own email with nothing but a real certificate (maybe that matters) and SPF with no (delivery) issues.


> I have just given up running my own mail server after more than 15 years because of this problem. Five years ago I had no problems, but these last 12 months have made me decide to give up and just pay.

I'm tempted to do so as well, but I haven't found a provider that will just use my domain and let me set up aliases as I can now in postfix. (Wildcard on the domain to me, a handful split out to separate mailboxes for scripts to chew on, etc)


I configured a pretty simple system (dovecot + opensmtpd) following the guide on the Arch wiki in under an hour, it really wasn't that tricky. I have dodged some common problems, though:

- my domain wasn't completely unknown so I had no issues with gmail flagging me as spam

- I use my gmail for signing up to things I don't care about mail from, so I have yet to actually receive any spam (at which point I will figure out how spamassassin works)

I don't think we need any additional software to make the process easier, maybe just better documentation from the distributions themselves or some already set up bundles. Hosting providers like DigitalOcean already have ready-to-deploy images for WordPress and Docker and whatnot, why not one for mail?


Running a mail server is easy, providing a mail service is HARD. Deliverability is really difficult. Being online all the time to make sure you're always recieving email is hard.

I was constantly plagued by worry that the important email I sent never actually arrived, or the important email I was waiting for had been sent but I never actually got it.


SMTP requires the sender to retry delivery if the destination is unreachable.

https://tools.ietf.org/html/rfc5321#section-4.5.4.1


Just because it's in the RFC does not mean it will be obeyed.

I tried running postgrey - a system that rejects senders it doesn't recognise, tells them to "try again in a minute", and accepts the second email. The theory being that spammers rarely check for these responses because they operate on a "fire and forget" basis. In reality it didn't work because lots of email services didn't correctly respond to the "try again in a minute" response - which is in the RFC!

Maybe it's an extreme example. You expect an SMTP server not to be up all the time, you don't expect them to randomly reject mail and ask you to try again later, but you get my point. It just adds to the stress/worry. I found myself default to Gmail a lot of the time for this reason.


I don't know if it still exists, but I'm sure it could be found or would make a good service. I seem to recall paying about $10/mo to a company that would provide secondary MX services. They did no delivery/webmail etc, it was purely "You can list us as a secondary MX and when yours is online we'll forward".


* Mail you want to receive: Your server can be down for a few days (if I remember correctly, 5 days, according to postfix defaults) and still receive all emails when it goes up again. More downtime and the senders will receive permanent error bounces.

If you fear longer server downtimes, you also have the option of secondary MX servers.

* Mail you send: Just display the mail log (/var/log/mail.info) and see if you get 250 OK responses when you submit messages. What happens next is not your responsibility anymore (except for spam filter rules of course, you need spf, dkim...)


Great, now I've got to run TWO mail servers. That'll decrease my anxiety levels. ;D


I configured my server once, over two years ago. I logged in on root once again, when I run out of disk space, I made a cron job to remove oldest logs monthly. It's centos6 and it works. Should I care more about it? Anything I should pay attention to?


It's not that bad: "apt-get install mail-stack-delivery" on Ubuntu and you have everything you need. Ok, to make it real nice will take some work, to get approved certificates, SPF and DKIM working can be a challenge. But in general you can quickly get going of you want.


Have you ever run your own mail server before?

edit: I've never spoken to someone who has and says it is 'easy'.


My main email system for me and my wife is an Ubuntu 14.04 DO droplet where I used said command to get going. I use ownCloud for card/caldav and ownCloud mail (and sometimes roundcube), my wife uses OsX mail. We both use Android mail clients. There is pain sometimes, when Outlook.com makes my mails vanish and admittedly I have yet to receive spam and have no system in place. This guide helped me a lot [0] there is way to much detail and a lot I didn't use but you'll learn a lot.

For more than I year now all I ever did as maintenance was apt-get update/upgrade.

http://arstechnica.com/information-technology/2014/02/how-to...

Ok, it did take me a while to find mail-stack-delivery but it really is a good basis.


I'll back up the downvoted parent comments claim that it's "not that bad". It's caused me a handful of stressful days over the last ten years. I don't have problems with deliverability since I implemented DKIM (as far as I know anyway).


It is easy if you don't care if only half the people you email get your email (i.e. you are a spammer).


I only occasionally had problems with outlook.com, I contacted them, their spam system is an outsourced blackbox. They advised me not to send mass emails... Which I don't of course. Gmail and Apple never gave me any problems. Yahoo banned me after my first mail, no idea why but it seems to work now. I feel like I'm fighting the good fight (decentralize all the things!) so I'll just accept the shortcomings, which are in general, not that many. There are even some nice things like I can have mail addresses that go to both me and my wife, unlimited spam addresses etc.


I'm running my mail server and it's not that hard IMO. There are a lot of guides, standards are easy to understand. I can't say how much one would devote to setup his server, probably few hours should be enough.

Gmail marks my mail as spam, for new recipients, though. I don't care, every gmail user usually checks his spam anyway and when I do care, I'll call person to ensure he got my mail, regardless whether I use gmail or my server.


>I don't care, every gmail user usually checks his spam

So you need to start a business where every hour waiting for a response means money lose for you.


Agreed. Ran postfix + cyrus + DSPAM for years and dreaded dealing with spam every day--couldn't get DSPAM to really work reliably, so had to fish out ham daily, and mark a lot of spam. Never really invested the time to learn the ins and outs of postfix hardening, nor to figure out cyrus quirks that kept bugging us on a low-level basis.

With Fastmail for the past more than year, and very happy with them. I'm also very happy they're clearly at the forefront of OSS mail system development.

(Don't use CarDAV or CalDAV, so don't benefit from their efforts there.)

(Though oddly enough, just today I started getting a bunch more spam. Maybe there's a Christmas rush of spam, so the .01% that gets through is more noticeable?)

In general, I haven't had to go fishing for ham more than once or twice in the past year, which is pretty amazing.


I might go that route myself. I've been running my own email server since 2012 and I'm just tired of having my stuff constantly flagged as spam


use rdns, spf and dkms, luke.


... and sign up and monitor every FBL you can get your hands on, have outgoing spam filtering gateways, procedures to rotate subnets in case Microsoft/Outlook/Hotmail receive a single complaint about any message relayed by your MTA and take their sweet time "conditionally mitigating" you. Pray daily that none of your users get their email creds stolen on a coffee shop wi-fi.

It's a very lame part-time job.


you won't got onto a FBL if you are the only sender.


Make that two of us!


> In future we plan to have a third separate addressbook for automatically added email addresses.

I just want to say that this is something I really want. I don't even care if it's exposed over CardDAV, I just want to get automatically-added addresses out of my normal user addressbook. A couple of weeks ago I realized that my addressbook had a bunch of contacts in it I didn't recognize, so I cleared them all out. Today I checked and there's 3 more unwanted contacts.

It actually feels very broken to me that FastMail is automatically adding things to my address book without asking me, and without any obvious way to turn it off. I don't mind it recording this stuff in a separate addressbook to use for spam filtering and for auto-filling during message composing, but it should not be part of my normal addressbook and should not be coming down over CardDAV for my user addressbook.

The reason this is a problem is because I use my FastMail CardDAV addressbook as a shared addressbook for me and my wife (she doesn't use FastMail herself but I do), since we use iCloud for our normal address books but iCloud doesn't have a way to share addressbooks. It occurs to me now that since I'm actually using a Family account (for a single user, so I can have a custom domain without having the more expensive plan), I have a Shared address book available and I should just switch over to that (since I assume nothing ever auto-adds there). But it was still a rude surprise to find a bunch of unknown contacts appearing in my Contacts app on iOS.


You can turn this off in Settings -> Preferences -> Auto-save contacts.


Good to know, thanks. I'm still looking forward to being able to have auto-saved contacts in a separate addressbook though, as I do like being able to autocomplete addresses when composing messages even if I don't want that address in my contacts list.


Fastmail is fantastic and their policies and pricing are fantastic as well. Really happy with them, and moreso after this story and the recent DDoS troubles they went through.


> Even if we reported a bug to Apple right now, it would still take months at a minimum before a fix is in the field and on all devices. So we need to get this working for everyone so we can move CardDAV out of beta.

Basically make a hacky workaround, and leave the situation as is for everyone else.

This is not meant directly as criticism, but one thing I really appreciate about the OpenBSD team is how they refuse to "fix" things by creating bad code as a workaround to other broken code.

They always submit patches upstream, and everyone gets to benefit.

If more people had that attitude towards code quality, we might not be in such a buggy universe as we are now.


I've had some deliverability issues with Fastmail, and the search feature is horrible. Other than that, I've been a satisfied customer for over a year now.


I've actually found the search in the web interface to be very fast and pretty good in terms of finding what I'm looking for with >70k messages.

When I switched from Gmail a few years ago (https://masnick.org/2013/07/19/fastmail/), one of the most striking changes was how fast FastMail's interface was compared to Gmail. FastMail did a huge update to their search code just as I switched, and I haven't looked back.


Can you use multiple domains with fastmail? And do they allow wildcard addresses/forwarders?

It's probably on the website, but I haven't found it yet.


Yes and yes. I'm using those features since I migrated my mail there from personally hosting an all forwarding config.


Cool, thank you for the response! I'll probably give it a shot for my non-gmail.com emails and see how it goes.


Well done guys!




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: