Thanks Bret, I appreciate you taking the time to dig into this. You're right, my complaint is the disconnect between the terms-of-service and robots.txt, so I'm very happy you're addressing this.

I just wish we could have had this conversation a few months ago, before you guys threw your lawyers at me.