Hacker News new | past | comments | ask | show | jobs | submit login
Back That Gmail Up (jerodsanto.net)
106 points by sant0sk1 on May 27, 2012 | hide | past | favorite | 66 comments



I'm not leery of Google shuttering its Gmail service, but I can imagine a scenario where they cut off my access to Gmail and I have no way to plead my case to the search behemoth.

I am baffled at how people continue using such a service for something as important as e-mail in 2012, while still being fully aware that their access can be completely cut off at random with no reason or recourse. Does everyone just assume "it won't happen to me"?


I could die while driving--it happens tens of thousands of times per year--but I still drive places because it is much more convenient than the alternatives.

Gmail is very convenient, so I use it despite the very small risk that it will backfire on me.


> I am baffled at how people continue using such a service for something as important as e-mail in 2012

It's free and works PERFECTLY 99.9%+ of the time, for 99.9%+ of users. The edge case (being shut off) can be easily worked around by having backups and changing your MX records to some other email provider ifwhen you find yourself without service from Google.

This risk is acceptable when weighed against "worst-case is 24 hours of no email while I switch providers", and even that probably won't happen. If it does happen to you, it's just a small temporary inconvenience with no lasting effect.

"Shit happens, always have regular tested backups (gmail or otherwise)" negates any potential data loss.


Of course that only applies to Google Apps and not @gmail.com addresses.

But good point, and I reckon that's another good reason for anyone using email for professional purposes to get their own domain and use it.

(One of my pet peeves is seeing businesses advertise a www.mycompany.co.uk website and a bob@aol.com address. Small places do it a lot - why would you buy a domain and not use it fully? Why?)


Inertia.


Shit happens with every provider, free or paid; hoping it won't because you pay $400/month for it is foolish. The smart thing is to make sure it doesn't affect you much, by decoupling from the service and turning it into a replaceable commodity as much as possible. That means using your own domain, keeping backups, etc. Then, when shit happens, you can just sign up for another provider and keep going.


Shit happens, but with nearly every other provider, you can at least talk to someone to find out what happened and get your service restored, or at least get your data.


You can always get your data if you do backups - like you should in any case - so that point is moot.

So, you're paying and giving up on a nice webmail client with two-factor auth (how many providers offer this?) for "finding out what happen". I don't think it's a great deal.

I'm not a Gmail apologist, by the way - I think there are valid reasons not to use it, like not giving Google a copy of your whole life - but I don't find their policy for when "shit happens" a real problem.


You do not have to give GMail up, not sure where you read that...

The point was, that exclusively relying on it is not smart, if email is important to you. You can easily use the full benefits of your gmail convenience while still having a fully working fallback. There are even two ways of doing it, one would be to simply forward your email, or using your own domain on google apps.


You do not have to give GMail up, not sure where you read that...

Hmm, from the first post I replied to (by 'there'):

    I am baffled at how people continue using such a service


Because domains never get seized, of course. Nor are there ever shenanigans with transferring between hosts.

I'd bet that Gmail has better uptime and less chance of loss than owning your own domain, for most cases.


I couldn't agree more. Email is essentially the skeleton key to all your other accounts thanks to email reset, even if you use your own domain and don't risk the virtual armageddon of losing both all your mail and your address forever you still risk losing all your mail. Even worse, if your account is hacked you have no effective recourse whatsoever and someone will have access to all your accounts, contacts and email history. I pay for Rackspace mail because when I have a problem, I can call a real phone number and talk to a real human whose interests (company continuing to be paid) are aligned to mine (having a working service).


If you use Google Apps with your own domain and make backups of your mail, then you have a very small chance of losing your address and your mail.

People hate on Google Apps, but you don't give up control if you follow both these steps (which is easy.)

I also don't see any reason why another email provider won't do what people are afraid Google will do.


> People hate on Google Apps

Really?

Do you know of any other (free) options out there with comparable options? I really only use the hosted email, but other features are always handy


> I couldn't agree more. Email is essentially the skeleton key to all your other accounts thanks to email reset

That's why I decided to get my own domain and use google apps 4 domain. I mean, who knows what happens in 10 years, maybe twitter will became internet-giant and I'll use twitter apps 4 domain then.


> I am baffled at how people continue using such a service for something as important as e-mail in 2012, while still being fully aware that their access can be completely cut off at random with no reason or recourse.

I am consistently baffled that people with higher than average technical skills assume that everyone is at their level.

Many people worry about Gmail/Google cutting them off with no/ or little recourse. The mistake in your assumption that I'm pointing out as an example is that most people don't know how to run their own mail.

So, no: I don't think everyone is assuming it won't happen to them. They just don't know what to do, and that's no reason to cast Gmail users as afflicted with the 'psychological law of self exception' (or willfully ignorant).


I think about this, but I find it far more likely that my domain goes bad for one reason or another, or my server crashes than that gmail shuts down my account.


At least you can just change some DNS records and get your mail working on another server. If your gmail account gets suspended there's no way to get that address back, ever.


That's exactly why I use the paid version of Gmail (as a part of Google Apps) with my own domain name.


What is so important about email in 2012? Tell your contacts your new email, and change restore email option at web resources. Sure, it is a bit uncomfortable and will take 1-2 days to finish, but probability of this scenario is very low.


It's also the backdoor to nearly every online account you have, by way of 'reset password' links.


That would be a problem if your Gmail was hacked.

The GP is talking about Google themselves blocking access. I'm not worried that Google will reset my passwords in other accounts.


The problem is, that it prevents you from resetting passwords, too. Which could very well lock you out of your own accounts.


For me-GMail is far and away better than any email and email interface offered. No other spam filter is competitive. Not other interface, web or desktop, is competitive.

To me, it's worth the tiny, tiny risk that I might get cut off.


Instead of putting the password in as a plaintext argument, you can store it in the OS X keychain and access it with the command described here: http://www.maclovin.de/2010/02/access-os-x-keychain-from-ter...

I use this in my .zshrc to create Bitbucket repos on the fly:

  function bitbucket() {
    curl -X POST -u masnick:`security 2>&1 >/dev/null find-internet-password -gs masnick.bitbucket.org | cut -d '"' -f 2` https://api.bitbucket.org/1.0/repositories/ -d name=$@ -d scm=git -d is_private=True
  };
(Running "bitbucket newrepo" will create a new repo in my Bitbucket acct called "newrepo".)


Another good one is http://gmvault.org. I used this just recently to backup and transfer all of my Google Apps emails to a separate gmail account since I'm transferring my apps domain to a different provider. Going just off of what the article says about BaGoMa, Gmvault seems to be comparable in features.


From a quick glance Gmvault looks preferable from a security standpoint as your credentials get saved in ~/.gmvault rather than being passed as a command-line argument, which could be viewed by any user on the system.


Been trying to get gmvault to do at least one successful run since it showed up on HN maybe a week or so ago. It always stops somewhere at mails from 2008 with no error message, just dies. Guess I'll wait until it's a bit more mature. I only have 6GB of mails too.


Gmvault can fail for multiple reasons, for example Gmail can cut the connection and throttle you for a while. There are also bugs left or GMail emails that cannot be downloaded (email corruption ?).

For this reason, there is a --restart option for the sync and restore mode (sync being the backup). The restart mode is a resume mode that will restart the backup or restore near from where it failed.

With that mode you should be able to download your 6 GB on disk.


My main problem with this is that it would require me to store my password, in clear text, in a shell script.

I'm no security expert but that raises a red flag for me. I'd rather have some way of entering my password each time I want to back up, but this would also make the cron process impossible.


See gmvault as mentioned in the comments here, it just saves an oauth token. Still can be used to access your mail but at least it can be easily revoked and isn't your clear text password.


Just chmod the shell script to 700, so that only your account can read it, together with an encrypted HDD that should be quite save.


If it's a shared box, you'd still have to hide the --pwd argument from `ps` process listings. You'd have to do something like make a ~/.gmail-pass file with the content "--pwd foobar", and call the script like this:

  cat ~/.gmail-pass | xargs python bagoma.py --email=foo@bar


Good point, hadn't thought of the process list.


https://www.backupify.com/ is a pretty good alternative as well


This may be a naive question, but why is it not enough to set your mail client to save a copy of all messages?


The problem which might arise is also, that being locked out of your email account will most likely result in being locked out of any "recover password" functions anywhere on the web. Thus, you need a backup of your email and preferably a working email address, too!


For many people, Gmail is their email client...


For many, it might be easier to run a mail client than a python script though.


My Mac app, CloudPull, does something similar. CloudPull backs up Google Mail, Contacts, Calendars, Drive (formery Docs) and Reader to your Mac. http://www.goldenhillsoftware.com/

CloudPull stores your password securely in the Mac OS X keychain.


I looked through your web site and one question remained unanswered:

Does it also do cloud push? In other words, can I use your app to restore the backups to a different Gmail account later?


No, CloudPull does not restore directly back to Google. You could do this independently by restoring to Apple Mail, and then importing into Gmail or another IMAP account.


Two problems with that:

1. I'd rather have one app do the whole thing, so that I know it's tested and it works.

2. How about labels, contact, calendars, etc?

I'd pay for this app, but only if it allowed full roundtrip.


Sorry, CloudPull has no ability to restore directly to Google. My emphasis has been restoring to local apps. In all cases, there are ways of importing back into Google, but none that the app does on its own.

The ability to restore directly to Google is on my roadmap for future versions.


Nice! That looks very well executed. Personally I'm glad to see you explicitly mention that it's read-only. Separation of concerns and all that.


So. GMVault looks like the best solution except:

1. It saves one big inscrutable .db file rather than individual messages in some compatible format or other.

2. It doesn't seem to play nice with laptop suspend. If I'm going to run it as a scheduled task then it needs to not get confused if I shut my laptop lid in the middle of a run.

3. It could be clearer about the correct workflow for continual incremental backups. It sounds like you probably want to do one full backup, continual partial backups and then occasionally a full backup in case your labels have changed. I don't like choices.


I am the Gmvault developer.

1. Gmvault doesn't save all your emails in a .db file. It creates a db directory which contains sub directories containing your emails in eml format. So it is using a standard email format and allow users to access individual emails.

2. I have already closed my laptop and reopen it and Gmvault restarted from where it stopped (tried on windows). Still there might be a problem so please report it following the instructions here: http://gmvault.org/report_pb.html.

3. Regarding the workflow, the recommended mode is to do a full backup and then daily do a quick backup. The quick backup is an incremental backup that only check emails for the last two months. It takes only 2 to 3 minutes normally.


Thanks for replying. I'm going to check again. I must have been doing something wrong. I'll contact you by a more appropriate means when I've figured out what.


On most stock OSes, this will show your password to anyone who views the list of processes on the machine.

Try 'imapsync' [1] instead, as it uses a config file for your credentials instead of command-line arguments, keeping the secret(s) out of the process list.

Also, the author of this blog post is showing his inexperience:

> BaGoMa's output is pretty informative so I send it to log a file in case I ever want to check up on it and make sure the backups are still working.

You won't ever do that (no reason to check the logs when it's working), and it will fail (effectively-silently) one day without you noticing, and then you won't have backups when you actually find that you need them. Don't do it that way.

Instead, use a cronjob wrapper script (e.g. [2]) that is silent on success and outputs on error, so the "cronjob-output-as-email" feature works to notify you when it breaks.

[1] http://imapsync.lamiral.info/

[2] https://github.com/sneak/misc/blob/master/cronify/cronify


> Try 'imapsync' [1] instead, as it uses a config file for your credentials instead of command-line arguments, keeping the secret(s) out of the process list.

Given where he said the box resides, I'd say he's got a major problem if unfriendlies are in s position to run ps -eF on it, no matter what is in the command strings, LOL!

And, if they can run ps, they can more than likely read the config files, too!

> Instead, use a cronjob wrapper script (e.g. [2]) that is silent on success and outputs on error, so the "cronjob-output-as-email" feature works to notify you when it breaks.

Excellent suggestion.


gmvault opens a browser window, where you confirm you want to give an auth token to gmvault. All without a password anywhere.


If you don't want to use the oauth token mechanism or if it doesn't work, you could also use your gmail password but it isn't recommended.


My point was that I don't want to use the password. Don't want it in my cronjob file, don't want it in my logfiles. It's really bad practice of programs to rely on the Password. On the bright side, gmail also allows you to generate application specific passwords.


imapsync doesn't work perfectly with gmails imap server. There are multiple scripts out there just to work around as many edge cases as possible.


"X It's hosted on SourceForge, just kidding that was weird and scary"

It is hosted on SF, and that IS weird and scary. In this day and age, why would your new cutting-edge FOSS offering be delivered on SF rather than github?


People were raving about it on a forum - well I'm sold on the idea. In all serious though, if you assume that your email service will be around forever you probably shouldn't be on the Internet. Cudos to the author, I haven't really seen the point stressed enough that you should backup your email, especially on a service as large as Gmail.


I like it. Just got it going.

Specifically I like that it stores all your emails as plain text, one file per email, so you can easily use/write other tools to search/browse your backups if you need to. Maybe someone already has?

(Well, it stores the raw email contents as text. Most emails have a text/plain bit but not all, so you can almost just use grep.)


Plain text one file per email is the standard Maildir[1] format. It's extremely well supported by email programs and utilities.

[1]: http://en.wikipedia.org/wiki/Maildir


BaGoMa puts your email in folders based on the first 2 characters of the file name.

eg 00/00a87... 00/00aa8... 0b/0b0a9...

So you'd have to do some folder shuffling before dumping the data into a Maildir server filesystem but that's not to hard.


If you're going to go through that level of effort and you're going to have a computer on at 3am, just run your own SMTP/imap setup and back that up.

You'll never have to worry about getting kicked off or them going under then.

I've been running my own mail server for 15 years and I've probably spent a couple of hours administering it over that time.


Then you just have a bunch of other worries; are your servers up and secure; not an open relay; has an anti-spam measure kicked in against your server so your emails aren't being read .... not saying you shouldn't do it, just that there are pluses and minuses to both.


Most of which is just paranoia and will never shoot you.

Are my servers up? Don't really care - SMTP will be redelivered if I discover they are down. They were down for about 2 days once thanks to the idiots at BT digging through the cable. No major effect other than some peace.

Is it an open relay? no it's not - I checked it.

Anti spam? I deal with it manually. I don't get much as I don't publicise my email address.


> Are my servers up?

Fair point on SMTP retries, but "up" also includes "not under attack, not already hacked, fully patched & secure".

> Is it an open relay? no it's not - I checked it.

Good for you, you clearly have the skills to do so and to be confident in it. Not everyone does or even knows what that means.

> Anti spam?

Actually, I wasn't talking about the receiving spam problem. You can deal with that on your own server in many ways.

I was talking about the email you sent being marked as spam problem, so your intended receiver never reads it. Maybe your reverse DNS records are wrong or missing, or SPF records, or one of a thousand little things. This is much harder to fix, as you have to do it on each mail host you send to, and worse, you may not even realise it's happening.

Again, not saying you or anyone shouldn't, just that there are downsides that need to be considered. I did it for years then realised my time was better spent elsewhere and moved to Google Apps.


> "It's hosted on SourceForge, just kidding that was weird and scary"

How the world has changed...


Does this generate some sort of "standard" mbox file? something one can open with Thunderbird or similar?

Probably more convenient than periodically running a POP3 client.


Great idea! I need to do this!




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: