Some time we'll get round to writing this up but there's a small customer of Cloudflare that gets a very high HTTP requests per second rate. It's a simple service (bit like a "what's my IP address" but not that) and it turns out that a quite popular hardware device hard-coded requests to this service and doesn't appear to cache the results and so it asks over and over and over again for the same information.
We've contacted the manufacturer and I think it's been patched but the life time of installed equipment is long...
This is similar to how Qualcomm's DNS servers got knocked off the air. An OEM shipped an update which would query a development TURN server we were running - once per connection, over millions of devices. It was a crazy day.
Are you not tempted to just block the requests from these devices, and let the manufacturer take the loss? I imagine serving all those requests is costing real money.
It's not a TOS violation. It did cause us some ops pain at one point (they were getting hit with > 50,000rps concentrated in certain locations). But one of the reasons Cloudflare can operate our service is we have 3.2 million customers who are doing all sorts of stuff. We get so much stronger from that great variety of traffic.
Could you be more explicit on the nature of the service?
I’d like to explore mechanisms for tests that detect IoT devices that misbehave this way (and other ways as well). Your anecdote sounds interesting. Is it unrelated to time servers? Unrelated to internet connectivity tests?
We've contacted the manufacturer and I think it's been patched but the life time of installed equipment is long...
Yesterday: over a billion HTTP requests...