That way it's the bill-payers that notice and get in touch, not the poor tech on the out-of-hours rota.

Also, it only firewalled the customer's public IPs from the Internet - it didn't stop servers. That way when the account is settled there was often nothing for the techs to do.

Also it was only ever done after the bank reconciliation was marked as complete, just in case...

We relied on goodwill and recommendation of engineers for our growth, so were keen not to make their lives harder for someone else's bureaucratic (or cynical) mistake.

> Cofounder and former MD of Bytemark, a British hosting company.

I've heard the exact same Ad Words story a few times now, someone used their CC or had at one point in the distant past used with Ad Words and the system flagged it. Everything gone with no warning, no explanation and no way to complain. Even getting to the front page of HN didn't work =)

[1] https://news.ycombinator.com/item?id=32237445

I'm never going to blame someone for not trusting credit cards. Only if you don't trust them, you should offer a better alternative.

Google should know that credit cards can get leaked/stolen. Suspending an account without notification is not the right solution. At least give the customer a chance to defend itself, change payment methods or switch to a different platform.

"This credit card messed with our money machine, it and everyone connected to it are now permanently blacklisted everywhere on our ecosystem."

They genuinely do not care who did what and whose number it was and who got hurt and whether it was used with or without permission. That CC tried to exploit the mighty money machine and that's it.

I just can't trust Google for something like running production services.

I just refuse to use Google for anything other than mail at this point.

There, fix that for you. If it's mission-critical, there are better customer-focused solutions out there than Google. Despite their massive brain power, they just can't seem to get dealing with users correctly.

I don't understand why so many people find comfort in being able to blame a vendor for problems that could potentially end in mass layoffs.

  they just can't seem to get dealing with users correctly

Doesn't get more dystopian than this. Fuck Google.

How do you have so many employees and yet everything important is run by unsupervised computers?

Not sure if having an Android phone is safe enough for critical use, because it's entirely tied to a Google account.

$29/m + 3% of monthly spend https://cloud.google.com/support/docs/standard

Or alternatively $500/m + 3% of monthly spend https://cloud.google.com/support/docs/enhanced

We're a Google customer and of course we had our fair share of issues (btw: it's the same with Azure, based on our experience), and we always escalate to our account manager. That usually starts turning the wheels much faster.

Also, do not be afraid to ask your account manager for additional discounts. Everybody in sales has some wiggle room to get you a better deal. With Google their account manager actually suggested that we work through a reseller and we're getting % off the list prices.

Dealing with Google, Amazon, etc.. is not like dealing with code. It's not only transactional and you need to invest some time in the relationship even if you're the customer.

I worked for couple of really big enterprises, one of which had such a high commitment that Azure sent two engineers to sit with the teams working on their cloud. Highest level Enterprise support .. and yet for actual non-obvious problems it took them a month to reply with: can't help you with this, you must be doing it wrong.

This was techsupport tho, I'm sure the billing for multi-million commitments was just fine and dandy including wining and dining.

One other company was a big AWS user (and some Office365 so there were Azure pitches too) and GCP tried to get in with the business as well. When we went to their offices the level of patronizing smug that came from their reps was astonishing, so off-putting that I refused all further interaction with them as I wanted to barf.

Seek out resellers or providers who are closer to yourself in size. You want it to hurt a bit if they lose your business. Of course you can go too far the other way as well, if you're a large client of a small company you'll probably get good service but they will be living parasitically off of you which isn't healthy and they may not be able to respond quickly enough if your needs grow.

General view:

* AWS - Great customer service. Steak house level customer service when moved from cc billing to contract billing.

* GCP - terrible service regardless of spend and regardless of dynamic or contract billing. AM do not know products. AM's are equivalent of BOA branch employees that solve a problem by calling the same number the customer calls and give the phone to the customer who came to see them in their branch, except that the customer service phone is the same opaque process the customer is subjected to. The only GCP product that had good customer service was the original App Engine, supported out of Germany ( I think ). We could get in touch with engineers responsible for it over video chat.

As I've said and others have experienced, GCP's AM are killing their business and we've built up everything around AWS. There's no need to even learn Terraform, all of what we do are automated anyhow without it. CDK is enough.

Screw Google, screw Azure.

I read that as "g-host"

That...always sticks out to me, but, when dealing with the Google Education folks, there were some things that even resellers couldn't do, and I had to email people directly (via LinkedIn stalking) only to have my issue disappear a few days later with no reply.

So the "make sure you invest in an account manager" is not really an option at Google. AWS does this very proactively.

Unfortunately after 5 years, I can still see Google Cloud screwing over their customers like this. They seem determined to send more customers to AWS.

As a sales person I'm thinking sweet, I get to send a PO to SHI for example, I don't have to negotiate price, volume discounts, or go back and forth with legal departments depending on whos paper we used. You just need to approve and I get paid, probably quicker than if I used an internal deal desk.

-

I had amazing AWS reps, that I had a good personal relationship. Request your rep to visit your office, and have lunch.

Seriously.

The best reps I had from AWS moved to GCP, and while I couldnt move my loads from AWS to GCP, I still had a good relationship with them. One of the best things an account rep can do is understand your business, and, in my case, actually pushed for changes in the system to our benefit based on how we were using the system.

They have evolved a lot since that occurred, but a good rep is going to seriously push for your success. (They want you to be successful and increase your use! :-) and thats a good ting)

So, make requests for your reps to meet with you face to face.

Considering the problems are repetetetive, the legal documents can easily be templated and reused and the victim would not have to waste much time or effort in court. Scale the lawyers just like we scale cloud instances.

Maybe even have a way to programmatically sue based on automated downtime metrics.

https://www.google.com/search?q=restaurant+customer+%22terms...

I.e., basically meaning that Azure refused having any further discussions between then and the parent user, as attempts to resolve that specific case. Not that they dont't allow the parent comment user to talk about it with others on HN (or elsewhere).

=============================

Greetings for the day!

I have received an update from Account Research Team: As part of our strong commitment to the protection of our customers and our interest in preserving the quality and integrity of the Azure Marketplace, we perform supplementary reviews of accounts which may exhibit irregular or suspicious activity. Your account was selected for one of these reviews and after careful consideration, this account will remain closed.

Please understand that we keep security checks like these in place in order to protect the quality and integrity of the Azure Marketplace

=============================================================

The ridiculous thing.. my employer makes huge use of Office365? We signed up for Azure via a slightly different path with no issues.. but I know this could be pure chance it just happens to work as the service could be teetering on a precipice of arbitrary policy application and the rug could be pulled from under it any second with the same infuratingly cheery account deletion..

An account manager doesn't cost you anything - neither would a CE (or SE). There would likely be a minimum spend required, or if you're below that a commit, but there's no %age of annual spend charged for having an AM. In reality, and AM will save you money, as you can negotiate with them, whereas you can't with a website.

Google is at fault for how they handled it, but the parents comment is still good advice for anyone reading this thread.

I’m pointing to this as evidence https://cloud.google.com/customers

I feel like a lot of the problems that people run into with GCP are avoided with two main components.

Number one you should look seriously at the “enterprise” designation for different products and services if you are looking for long term stability and guarantees about the future (details: https://cloud.google.com/apis/docs/resources/enterprise-apis) and secondly you need to buy actual customer support as an add on if you want to avoid the “I had to hit the front page of HN to actually speak to someone” scenario https://cloud.google.com/support

the main problem with azure/aws/gcp is that their parent companies will track everything you and others do, and then close every related thing because they have enough market share to not give a fuck about you.

There are more than enough blog posts of enterprise customers running into random issues with the big thing, buying "big three, promise edition" isn't much of an assurance.

I'd suspect most of those customers have rapid migration plans written up. For a reason.

I can assure you from experience that most large customers do not. For either major cloud, most large customers will have adopted proprietary services like AWS Redshift or GCP's Cloud Bigtable or Bigquery. None of these have anything like the possibility of a "rapid migration".

Not sure what to say in response here…

I mean, it's an error.

Sure, it's infuriating, and in worst-case could financially harm (or cripple) your business. But Google didn't intend for it to happen. That makes no sense from a business perspective.

So, as others here suggest, you have to plan around the fact that mistakes like this can happen, especially when you're small.

In the car analogy it’s akin to looking the driver in the eyes and them signalling to you it’s safe to cross, only to hit the gas and run you over as you do. No amount of looking both ways before crossing would have saved you.

I would say the advice here is, wait until the car is basically stopped before you cross the street rather than trusting it will because you think all the signs say they will. I do think this is solid advice for the production systems as well. All the signs can point to yes, but making extra assurances to avoid the bad path is valid for something critical like your production systems (or your life).

you mean like an email literally saying they acknowledge the situation and won't suspend your account?

Most places outsource their support to global teams don't have access. If you have an AM , in many places, they receive the same notice you receive, also they will personally send it and address it to you with a reminder. They also have a lot more sway as the sales side of every company has the power so they can get around all the process issues.

As the other commenter posted above, if you can have an account manager where you have critical services, it will provide you safety at the cost of a higher spend or minimum spending amounts.

Edit: Had about $1mm in GCP spend and couldn't open a ticket to get windows running properly. It took a week to find the AM and a day after he had it fixed.

I'm not sure how successfully you've dealt with support with companies, but I have almost never been able to rely on the promises of a support agent, as they are simply not the ones actually making the decisions (in this case on whether your account gets suspended or not).

Regularly XYZ does not happen, or not exactly in the way you agreed. They cancel your broadband too late, or your internet starts a few weeks too late, etc. You can try to get compensation, or even sue, but some of the damage has already been done at that point. While this okay for temporarily overpaying broadband, this is probably not the case for a company's main production system, where the risk is very high. The company might not even survive to sue.

Credit cards are exceptionally finicky, very susceptible to fraud, and payment can often fail. They're great for one off purchases and services that don't matter if they don't go through - but not suitable unless you're going to check that payments are made, which the OP seems not to have done. Perhaps GCP notifications should be better (I've experience the similar scenario with AWS and everything got fixed within a day) - but surely its up to the customer to make sure they pay.

100% a mistake on my part of course; I emailed Linode, explained the above, and they said "sure, is 3 weeks time enough to sort it out? I've delayed any action on your account until such-and-such date but let us know if you need more time".

I've been a Linode customer for several years, paying about $100/month. Just a guy running some small things, hardly a big customer.

If Linode can do it, then surely Google can. Yes, you will probably have the occasional non-payment, but is kicking your people off your platform for a simple administrative mistake/mixup that much better? Accepting the occasional non-payment is better than kicking off people at the drop of a hat. I've since paid several more $100/month to Linode and everyone won in this situation: Linode because they kept a customer who will keep paying, and me because I could get my stuff sorted without having to worry about having my services cancelled.

Google's policy is simply short-sighted on every level, and harms Google's business interests too. The reason they can do things like this is because they have money to spare. Every business that's not swimming in cash like Scrooge McDuck treats their customers like this; the "big tech" company are the exceptions.

Fortunately they have customer support, and 5 minutes after emailing them they resumed my services again.

On GCP you're basically SOL

No, because Linode is using humans to interact with you. Google will not humans in those situation, their mantra is to automate everything so they can scale up with minimal human effort, even when it doesn't make sense to scale up that way.

> however we went ahead and made a manual payment covering all the outstanding amount + extra

(the entire last paragraph of OPs text has those and more details)

I suggest not using and then arguing with and against fantasized stuff when we are talking about a concrete problem for which we have a concrete description, no?

Given that OP paid, of what use is your objection about non-paying customers here?

You also omit this important part:

> They ... assured the project will not be suspended.

That's fair enough - somehow I completely misread/understood that part of the post.

But my main point was that you should never be using credit cards for services that you rely on - particularly when billing/payment is automated and hidden from you. Credit card payments and checks fail regularly (10%+ for recurring payment charges) for legitimate (you don't have funds) and less legitimate reasons (a banks AI system thinking the charge looks fraudulent due to something outside of your account etc). Vendors are likely to stop the service as soon as they can get away with so they don't loose money.

That's over $100,000 a year, at that point having a Key Account Manger (or Success Manager or whatever) to talk to should be the expectation.

At other companies where we've had an active and involved account rep literally just shot off one email and got back a "no worries, I put a block on the account so it can't be suspended without my approval".

I know in the past I've had personal accounts go a couple months in the red when a card expired and I wasn't staying on top of the emails.

These sorts of things are really only an issue with one specific provider.

I really can't understand why anyone would use any of their services. Support counts. It's one of the first things I look at, when evaluating any external partner. What does it cost, how long does it take for it to happen, etc?

Whatever anyone else says in this thread, that's what you need to determine for yourself. Whatever company, what is the support channel like, how responsive is it?

If you don't know? If you can't get immediate emergency response and support, and I mean immediate, then why are you even hosting there?

We have been running production on Google Cloud for six years. We are not a large customer: our spend is in the low five figures. We pay for a middle-tier support plan and support has been excellent during this entire period. We very rarely need to take advantage of it, since the underlying systems are extremely stable and almost never cause us any issues (compute engine, GKE, GCS, Memorystore, Firebase, CloudSQL, etc). Just thought casual readers of this thread could use a counterpoint.

I can speak to this because a few months ago when I started a project, I went with Firebase. At the time the rationale was, I need something more complex than Netlify/Vercel, but not AWS that will suck up all my time figuring out IAM roles and other dumb stuff.

It helped speed things up in the prototyping stage, but I will likely move my infra over to AWS soon.

This is total standard in AWS and the ticket for this on GCP has been open for many years now.

That is not backed by GCS buckets - if it were, I'm sure that I would have found this solution while searching the web.

But even if: I would like to e.g. keep the latest 100 images always. I doubt this would be possible with a simply GCS polcy without writing custom code or something like a cron job.

https://cloud.google.com/storage/docs/lifecycle#numberofnewe...

> GCR is deprecated and I'm using Artifact Registry. (sorry for the confusion)

You're right that this functionality seems to not be built into the Artifact Registry backend (and that's weird†), but it does still exist: see https://github.com/GoogleCloudPlatform/gcr-cleaner (found linked from https://cloud.google.com/artifact-registry/docs/docker/manag...), and specifically the `keep` flag for it.

Note also how the project is hosted under the GoogleCloudPlatform GH org. To me, when I see GCP projects that are set up like this (in the GH org but disclaimed as "not official" in the GCP docs), this suggests that Google engineers built it knowing it's a pain point; and those engineers will support it to the best of their ability in the capacity of being maintainers of this open-source project; but Google as a company don't want to officially support it (yet), and so your GCP support contract won't get you any business-level support for it.

It's sort of like how, in Postgres, there is code which is maintained by the Postgres maintainers, but which lives under contrib/ as an extension. It's essentially a lability-waiver for that component.

---

† I do have a guess as to why Google do things this way. At least where dev tools are concerned, Google seems to eschew the usual distributed-systems architecture for long-running jobs (of having a thin API client binary that submits jobs to a cloud-side control-plane daemon, which then drives the job forward, and which can then be polled/subscribed for job status by said client.) Rather, Google seemingly have a philosophy of designing local fat clients that reach into the cloud to drive backend processes as the "control node" for those processes. The Cloud Dataflow (⇒ Apache Beam) architecture is designed this way, for example. I believe it's the reason that the Google Cloud SDK ships with so many binaries — there are a lot of fat clients in there that actually drive logic, rather than just sending messages to daemons that drive the logic.

And, presuming developers are issued good workstations, I can see the advantages of this architecture. A local control node synchronously knows its own status, rather than having to poll for it; a local control node can use local resources (like how Cloud Dataflow can consume and produce local files on the ends of the pipeline with the same streaming efficiency as a regular CLI text-processing shell command); and an operation started by mistake, with local control, can be cancelled by just ctrl+c-ing the control process.

Depending on how you design the client, it can also "mandate manual usage" — i.e. ensure that the developer is interactively running the process for the process to proceed, and therefore that said dev is available in case anything goes wrong. (I've personally dropped [async daemon-driven] Continuous Deployment, in favor of this sort of "synchronous dev-workstation-driven deployment.")

I wish someone at Google would write up a paper on this philosophy; it's pretty clearly implicit in a lot of their work, but I've never seen it mentioned explicitly anywhere. (Maybe it's just one dev-tools lead who has influenced a lot of these projects, doing what they think is "obvious"?)

That is object specific though, so probably won't work unless I use exactly one image.

> derefr 3 hours ago | parent | context | flag | on: Tell HN: Google Cloud suspended our production pro...

> But even if: I would like to e.g. keep the latest 100 images always. I doubt this would be possible with a simply GCS polcy without writing custom code or something like a cron job.

https://cloud.google.com/storage/docs/lifecycle#numberofnewe...

> > GCR is deprecated and I'm using Artifact Registry. (sorry for the confusion) > > You're right that this functionality seems to not be built into the Artifact Registry backend (and that's weird†), but it does still exist: see https://github.com/GoogleCloudPlatform/gcr-cleaner (found linked from https://cloud.google.com/artifact-registry/docs/docker/manag...), and specifically the `keep` flag for it.

No, it does not "exist". It's either builtin and then it exists, or it doesn't. What you linked is a way for me to build it myself. And I found this AND this solution is even linked in the years old Google ticket. And guess what, they didn't build it. On AWS no problem. As I said: years behind.

Sure, I can setup my own cron job (or here cloud run function / github action). But that's not what I expect from a leading cloud vendor. This is not a niche feature!

Please don't defend it, Google doesn't deserve it. Credits to whoever build the 3rd party solution, but Google really failed here.

> Depending on how you design the client, it can also "mandate manual usage" — i.e. ensure that the developer is interactively running the process for the process to proceed

Just no.

Let's face it: Google is just too incompetent to do it. Not the developers there, but the company as a whole in the way it is organized. Even IF it were as you said and it would be a philosophy, they could say that close the ticket, but it's still open.

I can list you dozens of similar things with GCP and related Google services.

AWS, Azure and GC are all on the same level. Any of them gives you „the best“ technology.

This is recent topic and you can see that so many people prefer GKE. That's not my opinion.

Sure, any cloud works good enough.

True, but unfortunately they are literally the worst at customer service.

If I had an "established relationship with a sales / account team" then yeah, perhaps I would have been informed. Unfortunately my experience is that unless you're spending above some relatively high threshold (>$10k/mo? >$100k/mo? >$1m/mo?) then Google don't prioritise your business.

Let me get this straight. This company does everything they were supposed to, paid on time, and Google's own error cuts them off, and you are blaming them?!?

From your profile I clicked through to your LinkedIn via your website, and I think it would’ve been becoming of you to disclose that you were a senior engineering manager at Google Cloud up until earlier this year, given the context of the discussion and your stance on it.

Your parent said OP did what they were supposed to and paid, but got cut off anyhow -- to which you replied "You don't know that. Maybe that's the case (I'm sure it happens sometimes), but most likely it isn't."

I'm not sure how to interpret that other than you saying OP is probably not being truthful in their account of the situation.

I have my Hetzner account set up so they pull the money automatically (through SEPA Direct Debit). Not sure how it works with credit cards, but it's not like pull payments are not possible with Hetzner.

Ultimately its as bad as car dealerships to require a special connection to get basic tasks completed.

At certain scale (in my case, 5 figures/mo spend with a vendor and up), paying upfront is not something finance usually likes and NET-60/NET-90 is the norm.

However, I remember a case when I was the culprit - spam was being sent from one of my servers (one of the users had a common username and password, something I should have never allowed) and Hetzner notified me immediately, but they didn't block the server in any way (something I'd expect). They gave me some window for explanation/solving the problem. I was very happy with how they handled the issue - but of course others may have had different experiences.

I suppose billing works extremely well for most GCP customers as well.

As for me, I hosted a side project on Hetzner a few years back for some European presence. IIRC I used PayPal for payment and they didn't support PayPal recurring payment, so I had to settle an invoice each month. Used them for about two years without problem, then missed payment for two months during a chaotic period when I didn't notice their monthly invoice emails. I got exactly one email reminder about missed payment titled "Reminder" (yes, that's all, I got all their correspondence in front of me right now) before the final "Cancellation of Contract" email and termination of my account. At that point, politely pleading with them to reinstate the account was useless, I only got a threatening letter back demanding a wire transfer of the ~€60 I owed within three business days or risk being sent to collections.

Was I in the wrong? Sure. Were they friendly or patient? No.

Otherwise don’t take their money and advertise that everything will be easier and more reliable

That’s why NET30/90/etc exist — without an incentive to pay sooner, invoices will just sit around on the receiver’s books until it’s convenient for them to pay (e.g. to make their financials for a particular quarter look good.)

Invoices under contract law are like PayPal’s arbitration process: “always favours the buyer.” A seller would need an explicit pre-existing contract clause stating a due date for an invoice, in order for a due date declared on an invoice to have any force; by default, it doesn’t. And because invoice billing is something companies tend to offer mostly to companies they’re trying to stay on the good side of, they don’t tend to sit them down for a binding contract negotiation when doing so.

I don't think OP's post is about payment options but about problem handling.

Backblaze has previously locked our account on a saturday.

They claim we were putting too much pressure on their cluster.

They could've locked the 1! token that was causing 90% of the load, but they locked the whole account which was used by 10 different apps for a veriety of purposes causing unecessary downtime in some production workloads.

Also the workload has been quite consistent for days/weeks, so they could've detected it earlier and/or address it in a better way.

I mean it sounds like a dick move by Backblaze, but comparing it to what OP is describing is downplaying how bad this looks for GCP IMO.

Going through with official contact at GCP, being assured he did everything right and that it was taken care of - and still getting shut down. In my book that's just inexcusable - if you can't count on that kind of support to work then how can you rely on that platform for anything ?

They're cheap and reliable, that generally comes at the risk of early termination. I don't think you can have a cheap, reliable, unlimited service; pick two at most.

If everything is metered it seems kind of odd to complain about a lot of activity. Since it's literally the thing you make money with.

Minor nitpick, but this is categorically not true. As a customer, one who proactively worked with AWS AMs, SAs, and S3 engineers directly before making the change, I still took down S3 by just changing how new files were stored in our S3 bucket. Yes, this S3 bucket was at the time the largest bucket (based on number of files), but still not “simply unbreakable”.

I have a friend who volunteers as a webmaster for a charity org and they kept getting notices about their impending shut down of G Suite and the need to move to Google Workspace. The problem seemed to be a credit card and yet, when my friend was showing me on his laptop, there was no place for them to enter credit card information. They ended up losing all their email and Drive & Docs access for days and couldn't even get an email or phone address - paid or not - to address the issue.

After getting burned at a mid-sized business I used to work at with Angular 1.x and Google Glass, I learned that Google apparently sees business services as experiments rather than long-term investments.

For example, if you look at the Angular 1.2 docs, look at the bottom of the page here and you'll see Google included their copyright on those docs https://code.angularjs.org/1.2.27/docs/guide/ie

If you look at the sheer amount of emails they sent me before suspending the account, Digital Ocean did everything right: https://i.imgur.com/ah13bEA.png

They gave lots of warnings and also sent an email with the date the account was going to be closed a week in advance.

Not sure how to mitigate these risks for providers that only support one "project owner" with a single billing address :/

https://en.wikipedia.org/wiki/Bystander_effect

Having multiple people that receive billing notifications is strictly better than having a single person - what happens if that person is on vacation, sick, overloaded? No one even has a chance of catching this. If at least one other person receives those increasingly important emails, they at least have a chance to catch this before it breaks.

It seems to work—at this moment, anyway—even in private mode for me now though. ¯\_(ツ)_/¯

The real kicker is that they refused to tell us what terms we had allegedly violated because it’s “critical to maintaining our security systems.”

And they had “just conducted a manual review” and confirmed the suspension.

Then, after escalating further and a few choice words about how much I’ve spent on their platform, sure enough they conducted another “manual review” and found no issues.

I’ve really tried to avoid moving to AWS over the years, but I’m really reconsidering.

Working hours are relative. Our support routinely gets awoken at like 3am because someone in Singapore, where one of our larger customers has outsourced their IT to, just started their day and decided to follow up on a case.

Though I'll agree that weekends should be considered off-limits regardless.

Sorry but for me this doesn't cut it when you have staff and availability zones in every continent on the planet. It isn't some mom and pop shop in one town. It shouldn't be much to ask a company of Google's size and breadth to consider your local timezone for urgent communications.

I mean we're based in Norway, but what if we outsourced our IT to Chile? How would Google or similar know that they need to contact our IT guys during Chilean working hours, and not Norwegian working hours? If it's an invoice that's not paid, well that's done here in Norway though, so definitely Norwegian working hours there...

They proceeded to DDoS my droplet.

Only a couple minutes later, DO responded by disconnecting my droplet from the Internet for 3 hours, supposedly to protect other customers. Thanks, DO, you let the DDoSer win. My droplet was handling the attack just fine, it just made my IRC connection a little laggy.

As a $5/month customer, I was just small potatoes, for sure. But I had ideas for products in the future, and I decided that I would not use DO for them, knowing that some skiddie could just take it offline for 3 hours by just packet flooding it for a couple minutes.

Budget cloud hosting on a credit card allows scammers to use stolen CC numbers to spin up VMs to mine crypto.

Essentially the hosting company is offering a cash equivalent for unreliable credit.

This kind of abuse occurs at enormous scale, because it can be worth it for an attacker to use botnets to throw tens of thousands of credit cards at hosting companies. If you never actually expect to pay any money (especially your own!), then even thirty minutes of free compute is worthwhile to chase. There have been similar attacks against GitHub Actions, because they're also free and general-purpose.

This is also why the free-tier, dev-only, education, or similar MSDN or VS Subscriber type cloud subscriptions never permit the use of GPU instances. Too tempting a target!

The large providers like Azure and AWS are basically printing money, so they don't care about the small-fry scammers. They're too busy trying to hold on to the firehose of cash.

Smaller, budget providers like Digital Ocean are running too lean to tolerate scammers, but also can't afford to have humans in the loop from the sheer scale of the attacks.

So they use heuristics that are 99.99% accurate (or whatever), which means one in ten thousand legit customers gets axed along with the scammers. Oops.

Essentially, as a customer of these small providers you are taking on some of their risk in exchange for the discount over the big-name vendors. This is especially true if you pay them with a credit card, irrespective of past payment history. Like I said, they simply can't afford to keep a human in the loop.[1]

An example that came up here was a startup that had an account "ticking along" with a handful of dev stuff, went "live" with a big launch, and so almost all of their servers was loaded to nearly 100% capacity thanks to efficient containerisation and auto-scale.

Good job devs... except that looks identical to a hacked account that has suddenly started to mine crypto on every machine.

Axed.

[1] It's even more complex than you think. Employees can and have been bribed to let the scammers through! The employees themselves might be mining crypto. The scammers can trick them, lie, beg, or just figure out the system from repeat conversations. The only recourse is to take the human out of the loop entirely, nothing else works for them. If you get to be the 0.01%, you generally have no recourse.

Customer service representation is one of the biggest expenses these companies have. That’s why they go to great lengths to make sure you can’t contact them and why they just close your account with no recourse as soon as you start causing trouble.

When my account started spending 300-400 USD monthly, I was contacted by an account manager, who was at my office for a meeting two weeks later.

It doesn't matter. Google operates at scale. As long as they save more money by not bothering to handle such incidents better, they will see it as a perfectly good business policy.

Incidents like this turn customers off for life.

Small customers are completely interchangeable for larger service providers. You can easily get new customers by throwing around some free credits to college kids and posting a few tech ads masqueraded as tutorials.

If you have a spend of over $2500/mo you can apply for invoiced billing directly from Google or if you have less than that you can contact a GCP reseller partner to get invoiced by them.

Some GCP reseller partners also have a close relationship with Google and the contacts that come with that relationship.

Why should a business relationship with Google (or any other cloud service) be any different? I don't use GCP but if I was going to start, I would ensure that I have a person to call or its a non-starter.

Depends on the size of the account surely? Maybe there's some support person to talk to, but there's plenty of services I subscribe to fo $10-100/month and I dont have an account manager for any of them.

Google however is infamous for not providing support at all on all their services, even if your complete digital identity got killed off because some AI can't recognize that the "CSAM" you sent was in fact communication with your child's doctor [1].

[1] https://www.heise.de/news/Nacktscanner-Unbedachte-Fotos-vom-...

Because google choose to make this as difficult as possible. A couple of years ago I worked on a third party IAAS database cloud product (think Atlas or Elastic Cloud, but for a different database), and while Azure had the worst tech overall, Google were ultimately the hardest to deal with by a VERY long way because they don’t acknowledge the existence of people as anything other than ad targets.