Literally just sent an email to my devops guys to move off cloudflare asap. This cavalier lack of respect is a diservice and insult to all the people who rely on my product for their livelihood.
we use cloudflare for their dns so not a complete change of arch. (this kind of lockin is precisely why I've stayed away from faas)
more importantly, its important to send a message. We depend on these services for our livelihood. if I'm paying for a service, the least I'm owed is the ability to get in touch with a person to rectify the situation as soon as possible. Companies who want other companies relaying on their service need to provide that if they want to be taken seriously.
EDIT: also, not to knock jgrahamc. appreciate that you're looking into this but one person on an email is not a scalable customer service solution for B2B. at the very minimum, there should be some sort of platform for filing the tickets, getting a timeframe on resolution as well as options to pay for faster turnaround.
Totally agree they should have provided a warning. Any Saas (and really would apply to social media, web mail, etc) should clearly warn before taking drastic action if possible. But I don’t see how you’d have dns without vendor lockin. It’s not like moving dns to godaddy would be better. Route53 is nice, but aws is also a vendor lots of people are locked to
Similar for ddos protection- you almost have to use somebody.
Had similar issue in recent past, and was able to get it solved via the Discord channel. From what I understood is that it requires a manual over-ride on R2/ Workers because the thing that checks for the 2.8 TOS violation is not able to see the difference between Workers/ R2 and the standard CDN service.
If you go to the R2 Discord channel you see this happening every other week.
What is also kinda annoying is customers can't create support tickets because it requires a plan. Which imo is bad given these customers pay for R2 and often have a ton of data on it (which is why the 2.8 gets hit...).
Hopefully you can get this fixed permanently (for all customers at once, and not case-by-case).
Dear John, I absolutely love and admire Cloudflare’s services as a customer (and recent investor), but please please get stuff like this sorted as it will absolute ruin Cloudflare’s reputation in the long run. I beg you!
This happens all the time though. Your business and processes are fundamentally broken.
And all you do is pop up on HN anytime someone complains, that's enough of a red flag to avoid your business completely and actively keep all my clients away from you.
Worrying part is the account could get banned. Could you reply here if that’s true? Would that take down dns/registrar etc functionality? Google does this and people would move to cloudflare thinking you wouldn’t do this account ban but nowhere is it stated.
The fact that you just post your e-mail address and invite your (likely many) customers here to reach out to you would probably surprise me from another company.
I've been doing this long enough that just about every major vendor I've worked with has had (and taken) the opportunity to disappoint me with some unreasonable decision/change and even an occasional (unwarranted) account suspension. I think I've convinced every customer I've worked with to purchase a Cloudflare subscription. I've worked with support once and I've worked with someone handling the beta testing for Warp (a Romanian gentleman -- he called me and shipped me a T-Shirt).
The two people I talked with didn't have to tell me they enjoyed their job. You could hear it in their voice. The guy I talked to about Warp was as far from a salesperson as someone could be, yet he couldn't help explain some of the details about how interesting of a product Warp is.
I can't count how many times I've pointed people at the Cloudflare blog to learn about "how all of the stuff between your code and the user's browser 'works'". I remember reading a post several years ago thinking "they're basically explaining how they achieved a major competitive advantage well enough for a competitor to duplicate." I didn't think that it was a bad idea to do so -- realistically, it didn't represent a loss of IP -- I'm just surprised so much energy/time would be spent writing highly technical posts that sometimes "give away secret recipes" in a sense. It's wonderful from where I sit.
I expect the HN crowd will recognize that people who have a problem/issue/incident with a company/product are a "flobbity-jillion" times more likely to write a post (and have it hit the front page) than a guy like me who's had 30-ish opportunities to integrate your products into things I've written and have been delighted every time.
... Cloudflare has a lot of customers[0]. They have to balance the cost of providing (a lot of) human support against the cost they can reasonably charge for their products. It's a balancing act, and one that has worked out well for me, personally. It sounds like this issue is happening related to R2, which is quite new.
You're not likely to see a post hit the front page with the title "I've integrated Cloudflare's products with 30 or so customers and never had an issue" (or even be written). But experience an issue this large and you're going to do everything -- make calls, post things to social media, reach out on HN where you know the CTO is an active participant -- and a lot of those are going to get attention from the small percentage of customers who felt wronged by CF but hadn't spoken up.
It's a crappy situation because it gives the impression that things are a mess when -- I'm willing to bet -- it's something along the lines of a problem in a quota checker and a failure of internal process to escalate the problem appropriately. That happens at every big company in various places all the time.
Really, the only major difference here is that unlike every other big company, their CTO actively watches Hacker News. When a problem pops up, he willingly chooses to be Customer Service and from the sounds of it, that escalation to address "problems like this" is now happening. There's going to be gaps like this at every company. When I worked at "BigCo", if something like this hit the front page of HN, you could expect a mess of people to have their phones ring. Work would be done to respond to the customer (variations on "acknowledge/minimize/suppress" communications -- on official company hosts). Staff would be forbidden from interacting in the ongoing discussion. The CTO might have had to have explained to him how to get to the web site containing the complaint.
[0] I don't work for them; I'm just a happy customer so everything here is my view from the outside.
I wouldn't say it indicates there's a mess, just that there's clearly some sort of broken process somewhere.
1. The customer is deplatformed without any notice
2. Customer support is failing to act on a false positive in a prompt manner and the customer has no recourse but to kick up a stink publicly
Both of those are fixable problems and I agree that it's generally a positive to see a company's CTO act in so public a manner. That doesn't mean they shouldn't try to improve things from an internal process perspective though.
There's a number of users in this thread who describe being "ghosted" by your sales team, including for tens of thousands of dollars per year subscriptions. It seems like the email responsiveness you're personally offering does not match with what some people experience from Cloudflare in general, so I'm not surprised people wouldn't think to email you and expect a response.
I have a question that I couldn't find on the help docs: I got several domains on the early bird Pro price. Do you plan to discontinue the Early Bird pricing this year with the pricing increase?
Money is not free, the cost of money has gone up considerably in the past 6 months. I haven't seen any indications that money will be cheaper anytime soon.
All "VC funded" "free tier" and the like will be put on the back-burner. If you know anyone with a small datacenter and a decent peering agreement (3 lines of at least gbit) now would be the time to kick money their way, and tell everyone else to.
It was tough times for small companies these past several years. Imagine trying to compete with netflix when their price was "all you (and everyone you know) can eat MP4s for $8". I actually cancelled my netflix subscription as we weren't using it anymore and the price was creeping up faster than siriusXM subscriptions.
I know this is edgelord to post on a VC forum, but I haven't seen any indication i am wrong yet. Big news is 80,000-120,000 tech workers being laid off by the big 10, but what about all of the layoffs at smaller companies that are VC funded? What's that number look like?
sorry, I did HN post in desperate move so my service can be online again - did not try to blame anything on @jgrahamc - other than that incident I'm very happy with Cloudflare Workers, it's an awesome tech.
This is the height of knee-jerk reactions, worse still it's largely pointless. Unless you're big enough to negotiate a specific contract with a cloud provider you're always going to be at the mercy of their catch-all policy.
The only way to actually be protected in this case is to run a multi-cloud strategy. Even then it's only going to protect you so far if you piss off the powers-that-be / community (see the hosting trouble Parler had as an example, not that I'm fond of Parler or anything).
If the redundancy is already in place to not fully rely on cloudflare's product (whatever it is, DNS, R2, etc) the it's not a kneejerk reaction.
It's an "I don't want to wake up to all our stuff running only on the backup provider because cloudflare shut us down for seemingly no reason with no warning".
It's avoiding unnecessary alerts and triage for the ops team by snipping an apparent liability from the stack. I've already done the same after seeing a few of these kinds of interactions with cloudflare in the R2 discord.
When I see a blog post detailing why this has been happening so often, and what they've done to fix it, I'll happily pull that infra code out of the mothballs.
So instead you're running only on the backup provider now. Congratulations on invalidating your multi-cloud strategy. You've failed to understand that the core reason you had that strategy is that any of them can let you down.
Every single one of the cloud providers has had instances of this kind of problem. It's somewhat an inevitability of the way they all work. Eventually someone triggers an automated system somewhere and gets taken down. Or has outages that they shouldn't have had.
Better cloudflare where the CTO hangs out on HN, than Google where both the ban and the appeal are not even humans with empathy.
Not OP, but we run two environments each of our service on Cloudflare Workers, Deno Deploy, and Fly.io (a small service albeit, 200 to 300 requests per second). In the event one is down, we switch to the other (via DNS).
