> Until this change, customers who have used fewer resources have covered the costs, in a way, for other customers who have used much more resources. We want to make things more balanced.
I may know one of the culprits -- whom I will leave unnamed here. But the company, who is fairly popular, built out their own CDN via putting a bunch of nginx caching proxies on various Hetzner servers around the world. It apparently was really cheap and very effective. Given that they were bootstrapped and this was prior to Cloudflare really being that popular, it was a great strategy. This was true like 8 years ago, so maybe it has changed in the meantime.
Sounds like a completely legitimate use though. Hetzner were widely telling people about that 20TB limit, so why would they be surprised when people use them as CDN boxes?
Where's the surprise? It's the classic business 2-step - drum up interest with "too good to be true" features, then cut them back. The marginal customers who need those features leave (and are too expensive to keep), everyone else is used to your product and stays.
Hetzner has been in this business for a very long time and I’m sure half the German small to medium sized companies use Hetzner. They have costs as well and with virtual servers they just assumed the usage pattern was similar to their eu data center and that didn’t work out so they increase the prices a bit. So that on average their calculation works out again.
Like, this is not Netflix tripling the prices over a few years.
Until you remember that marketing is a separate department from finance which is a separate department from ops/engineering.
The engineers said 20TB in aggregate was fine but likely didn’t consider the “bad apples”. Marketing obviously wants to use the biggest numbers and then finance comes in with the hammer and dev points to egress as an simple way to upset rhe fewest number of real customers.
It does make sense: The average was low enough that a 20TB cap worked. Then marketing started boasting about the 20TB limit and attracted a bunch of high-bandwith customers, thereby driving up the average making 20TB decidedly "not fine".
I think culprits is a poor choice of words since it means someone suspected of a misdeed. I could perhaps understand using it for example for someone that tried to store a petrabyte of storage on a consumer unlimited storage plan. But in this case Hetzner set a specific data usage amount you are paying for so using that amount is not a misdeed.
It also may depend on peering arrangements Hetzner has. If EU ISP more inclined to peer with Hetzner than US one bandwidth in EU will be cheaper for them.
- in EU there are large peering exchanges to swap traffic
- in USA no peering exchanges exists and you need to pay for your traffic most of the time. Few big operators in US and they enforce this.
Looks like some deal wasn't renewed and they lost a big chunk of cheap pipe or/and some of their upstream providers decided to do something with routing.
Also, Hetzner is way bigger in the EU than in the US. Good access to services hosted by Hetzner is thus more relevant to EU ISPs, because customers in the EU will probably use more services hosted on Hetzner infra. This gives Hetzner more leverage in the EU to negotiate beneficial conditions with regard to its uplinks and peering agreements.
probably streaming platforms have better content on the US and everyone wants to exit there? so they are mostly serving US traffic for several vpns all over the world connecting to CDNs in the US.
Isn't Hetzner impacted in Europe at the moment from a cable cut? Wouldn't surprise me if there is a wave of people moving stuff over to the US because of that. I don't know much about cloud though. I believe "The Ship has arrived and repairs are underway, which will still take some time." and "the repair may take up to two weeks.".
“Culprits” because they used the service they paid for within its advertised limits?
It’s the same with cloud storage providers. First give out a massive amount of storage and rapidly gain users, then cut it down after blaming people for “abusing” it. How about you advertise your correct capacity to begin with?
They are simply deflecting blame for their own enshittification.
Having said that, I am usually empathetic to these kinds of 'unlimited' deals because even though they aren't really 'unlimited', they do tend to be generous to the average use-case and average user .. Inevitably, and unfortunately, someone decides to test the limits and the entire thing collapses.
It reminds me of the Blockbuster "No more late fees" policy, which was a really good customer-friendly policy (speaking as someone who regularly returned rentals late) .. but then they were sued because an aspect of the policy had Blockbuster charging the cost of the rental to the customer if it wasn't returned in some period of time .. and because that charge looked like a 'late fee' they got sued. Urgh.
It’s not any more possible to correctly anticipate all pricing structure vulnerabilities, than it is to correctly anticipate all program and API security vulnerabilities. There is always a statistical chance of novel outcomes when humans are involved.
I was curious and checked if they are still using Hetzner. It appears not, so I can share who it was. It was https://artstation.com. Basically heavily oriented towards serving static images, so the CDN could have been really expensive. Doing a reverse IP lookup on cdn.artstation.com servers now resolves to Cloudflare and it has cloudflare headers on the response.
I may know one of the culprits -- whom I will leave unnamed here. But the company, who is fairly popular, built out their own CDN via putting a bunch of nginx caching proxies on various Hetzner servers around the world. It apparently was really cheap and very effective. Given that they were bootstrapped and this was prior to Cloudflare really being that popular, it was a great strategy. This was true like 8 years ago, so maybe it has changed in the meantime.