It's very easy to say the backup should be tested, there should be mirrored recovery systems etc. but this is a movie, even in a studio system this is a one shot deal, the systems and even the hardware will be disposed of after the master is cut,the people will leave for other jobs etc.
Now in the middle of crunch to make the movie - tell the producers you are going to double the HW budget for a test system to test the backups and you are going to stop doing any new work until a backup system is built and tested.
Personally I'm amazed they had any sort of backup beyond personal copies on animators machines.
"It's very easy to say the backup should be tested, there should be mirrored recovery systems etc"
I've been doing Unix backups for about 27 years. Back when it was more of a pain then it was in 1998 and certainly today. Total backup, incremental tapes the whole thing. Taking the tapes offsite. I wasn't a sysadmin but it was my business. So if I didn't have a backup it would be my problem and my loss both in work and money. I can assure you that setting up a backup plan (key word "plan") that evaluates the cost and trouble is worth it. The only reason higher ups don't approve of this is that they don't know about what can happen if something fails.
Ask yourself this question. Do you think Pixar changed their backup process after this event? I think they probably did. Not only because the failure happened to them but also because higher ups saw what could have happen if not for the woman who wanted to be with her child at home who saved the project in process.
That was my thought. Heck, it happened to me once.
Combination of a SCSI cable knocked loose, but not visibly so, and the brand new sysamdin (me in both cases) knowing enough to head and tail the backup log file looking for 'start' and 'complete' but not knowing to look in the middle for what was actually happening.
Lost a month of data in the (thankfully) little used legacy ERP system we used. If the entire plant had still been using it it would have been a disaster, not an inconvenience.
It's very easy to say the backup should be tested, there should be mirrored recovery systems etc. but this is a movie, even in a studio system this is a one shot deal, the systems and even the hardware will be disposed of after the master is cut,the people will leave for other jobs etc.
Now in the middle of crunch to make the movie - tell the producers you are going to double the HW budget for a test system to test the backups and you are going to stop doing any new work until a backup system is built and tested.
Personally I'm amazed they had any sort of backup beyond personal copies on animators machines.