Hacker News new | past | comments | ask | show | jobs | submit login

I did this to myself in college, thankfully on machines where I didn't have most of my stuff. I figured it was an annoyance and put in a restore request to the support group. Days went by and I was getting frustrated that they still hadn't gotten around to it. Finally I caught the head of the support group in the hall and asked about it.

"Well, the good news is that we're getting a new backup system." Long pause. "The bad news is that your files are at about the third gigabyte of a two gigabyte tape."

Ever since, I'm always frustrated by people that assume things are backed up. If you don't test your backups, you don't have backups.




I cannot say who the client is for this, but it's a government department.

I built a cluster which used a disk array for storage. When I was contracted to build this I was assured that they were building several other clusters for use as test, QA and prod and that all I had to do was build the dev system and document so thoroughly that the other systems could be built from the documentation. At the time I wasn't yet security cleared so would not be allowed to build the other environments.

Some time later (6-8 months) I was called back to help solve a performance related problem. When I arrived I discovered that the dev machine was now production, and that no other machines were built. I would have to do the work on production.

Worse though... their backup and failover mechanism wasn't what I'd documented, but involved an identical piece of hardware nearby which was powered off. They hoped to simply turn it on when production failed.

Their daily process was to approach the disk array and remove the 1st hot swappable disk from the live machine, and swap it with the 1st disk in the powered off identical system. When the array had rebuilt the first disk, then move the 2nd disk.

I am seldom speechless. It was only a critical part of a £2bn project. Beyond all of the obvious WTFs, I still wonder whether the 2nd machine could be powered on at all given that all of the disks were pulled at different points in time.


> If you don't test your backups, you don't have backups.

I couldn't agree more! It's amazing how often the question "we take backups, right?" produces "uuhh..". And yes, if you don't test them, they're worthless. Because the rule is, if you haven't tested them, they won't work when the shit's really hitting the fan.


Back in the day we would take the days backup tapes and restore them to a test system that was also available as a "playground".


How should I test my backups?

I have a projects folder, which is on Dropbox and additionally on Time Machine. I can confirm they are really in both by browsing them. Anything more I should do?


Take a fresh, empty system (a VM if you lack hardware); restore data from backups to it; check if you can do your work.

It may be that your projects folder really is backed up, but that it relies on some stuff that is outside the folder for whatever reason - for example, a data file that didn't fit in dropbox, or some changes to config files outside the folder (where noone remembers exactly what needed to be changed); or a connection that relies on a private key that's on your computer, gets used automagically, but is not backed up.

I mean, it's probably overkill in your case - most likely your project source is quite safe; but for a working production system it's an entirely another matter.


i especially love the (existing) backups that can't be actually used to bring the system "back up" .... experienced this more than once.


I guess there's the related one - mostly by managers and developers - where they feel safe if all the data is on a RAID array without backups. Arrgh.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: