Because of this, we're going to back up parts of our database more frequently. A lot of our data isn't mission critical, so it wouldn't make sense to do daily backups for it. Other things, like family accounts, obviously are. We're going to back those up daily from here on out.
Is making a backup of everything that costly? My experience is you'll screw something up trying to be selective. With storage being so cheap these days, it's hardly worth the risk.
I don't use mongo so it's not 100% relevant, but I found setting up a postgres slave and running my backups from that is a nice solution. There's no additional load on my master and I just dump the backups into S3.
I'm not too paranoid but we have, at least, transaction logs, replicated backup server, nightly offsite backup of transaction logs, and twice monthly full backups. That covers a lot of failure cases and isn't too much to transfer. Transaction logs are a godsend -- you could have even found out what you did wrong.
At work, our production databases are incrementally backed up every 2 hours with a full backup daily. The performance impact to the server is negligible and if little has changed the backup is small.
week? That seems kind of off. I'd suggest setting up backups more often than that, especially for production data.
Sorry to hear about that. I destroyed a drive once a long time ago (~1994) as well. Happens to the best of us.