Hacker News new | past | comments | ask | show | jobs | submit login

If it's already serial, and only works with one backend (rather than being an arbitrary mirroring tool like wget), then the Wayback server can easily "express its preferences" for rate-limiting by adding artificial delay to request-responses that pass the rate-limiting threshold. Backpressure shouldn't be the client's responsibility.

(It only is traditionally, because so many sites do nothing to protect themselves from "being too nice", so arbitrary-backend mirroring-client devs allow their users the option to ask for less than they want. This isn't a sensible protocol design, on either side; it doesn't optimize for, well, anything.)




Consider applying for YC's Spring batch! Applications are open till Feb 11.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: