Cool !! Is it robots.txt compliant? If not, it might be a good idea to make this available as an option/parameter.

For 'quick and dirty' tasks, wget -r can come in handy too.

Currently it doesn't look at the robots.txt file. I had taken note of this, and will be added in future releases. Thanks for the suggestion.

> 150GB and 100M records is not huge, not even close.

Close to what, if I may ask?

Agree. 1B rows is big. 100B rows is huge. 100M rows is large, but totally doable.

not even close to 'huge,' although my english syntax is a little rusty.

Regardless of your English syntax (rusty or not), what would you define as 'huge' for a MySQL app?

>Database optimization needs to generally take into account YOUR data and usage patterns.

I'd say this is quite true.

>I've found that an unfortunate number of people aren't aware of "EXPLAIN" and its use to help figure out query issues. Learn and use it.

Yes, we discovered this some years ago.

Thanks for you input.

1) Yes, and no 2) Both (3 servers) 3) We've analyzed/optimized this to the max (of our abilities)

I believe we did tune and tweak the heck out of my.cfg, but will investigate thread_dcache_size all the same.

I also believe we're archiving at least some logs (and such) to flat files already, but will re-visit this soon too.

Thanks for all your input.

Thanks for your input - and no, we're not webvet.com

ah ok, cool - good luck!

Hey, I'll have to check with the others here, but I think we have not been aware of 1. !!! Will certainly investigate this ASAP - thanks.

2. and 3. - already implemented to the max (of our abilities :)).

We did use a tool (perl script) called mysqltuner.pl on this sometime last year and it did give us some useful/actionable insight. Will check out what you've suggested too.

Thanks. :)

Will get the book and check out the blog soon. Thanks. :)

[1] Been doing already and [2] Will re-visit soon

Thanks :)