It sounds like they had their backups stored on a home machine and had to deal with a slow uplink to restore them. A cheap alternative is to use an inexpensive (but reliable) shared hosting provider for off-site backup. Don't use the account for serving web pages -- just use it as a fast, cheap parking spot for backup archives.

You're absolutely right. That's the plan going forward.

Are there any memcached-style tools that can sit between S3 and local servers? Seems like the most popular data could continue to be served up indefinitely.

I'm pretty sure that would defeat the purpose for many people. It sounds like in this case S3 was being used as a CDN (which it's not) to save bandwidth and improve response times.

Using memcached locally gives you neither of those properties.

Try varnish

Not sure. Would love to know if such a tool exists.

good lessons to learn.