It's not much better if you end up being a GCP customer.
My company has gone through 4 reps in 3 years. Every time we get a new GCP rep they just want to talk to us about "expanding our use of GCP offerings". The only thing they want to talk about is starting to use BigQuery - not my business at all.
I signed up for a Google Cloud Security summit, and afterwards a sales rep reached out me. It was obvious from the start they had no idea I was a gsuite or GCP customer. They then directed me to a NEW account manager (#4) even though I had been working with a different one. I had worked with the prior account rep going over our architecture to make sure everything was kosher (sustained use discounts etc). I even made them a schematic of our architecture on GCP at their request. Once I provided that to them I was met with radio silence.
It's really insulting, and to me obvious they don't care about my company at all. We're looking at other options.
I wouldn't be surprised if somehow this post leads to me getting an email from another rep "Wanting to start over and doing things right", which will inevitably devolve into the discussion of how can I use BigQuery.
I've had the same experience with GCP account reps. They always go missing and someone new emails us about how they are taking over 6 months later. Every call we have had with them has not resulted in anything meaningful. Our biggest issue is how their "highly available" Cloud SQL goes down every couple months for maintenance, not how we can use BigQuery.
> "highly available" Cloud SQL goes down every couple months for maintenance
Yep, drives me up the wall. We put a lot of work in making sure we're zero-downtime, pay for "Highly Available", and every couple of months I get the dreaded email. No explanation, nothing. Then we have to do the whole downtime dance.
If anyone from Cloud SQL is reading this, please, please stop doing this.
We’re currently migrating to Spanner for a variety of reasons - but the mandatory downtime on their Postgres CloudSQL offering will be the part I miss the least.
It’s insane that even with all of their HA and failover turned on they take the whole cluster down for as long as they like every few months!
Surely product makes these decisions, not engineers, right? I agree that customer empathy is important, but I don't think we can conclude that the engineering team (rather than the product team) is the source of the deficiency?
> Surely product makes these decisions, not engineers, right? I agree that customer empathy is important, but I don't think we can conclude that the engineering team (rather than the product team) is the source of the deficiency?
I haven't worked inside AWS or GCP, but I've never seen product get everything they want, especially around maintenance/downtime. If "less downtime" is on the roadmap but engineering is constantly pushing back "that'll be really really hard and take a long time and they're just using it wrong anyway," I can't imagine it getting done as quickly as at a place where the engineering team was also focused on customer satisfaction.
> that'll be really really hard and take a long time
It probably is hard and intensive. Engineering shouldn't lie and promise that it will be easier. Product has to take that engineering estimate and determine whether to work uptime or some sexy feature (and sexy features usually win because of perverse incentives).
Moreover, I have a hard time believing this for a couple reasons: first of all, I've scarcely met engineers who were opposed to improving product reliability, maintainability, etc. The portrait of Google engineers arguing that database services fundamentally shouldn't be HA (and customers are "using it wrong" for wanting HA DBs) is particularly incredulous. Secondly, I've never heard of an organization where engineering held political power over product decisions, but I have worked in several places where product dictated engineering solutions. Businesses trust product more readily than engineering because the things that engineering is always petitioning for are abstract and "costly" (deferring some immediate profit for reduced costs in the long run) while the things product wants are usually tangible and profitable.
Sounds familiar.
Product and sales areas of the business look at immediate revenues - and hopefully profit - but are always more keen to do that at the cost of increasing technical debt within the engineering side of the business. Unless sales see a real impact to technical debt, they will always choose the short-term approach.
Agreed, and I don't think that's even necessarily the worst thing. It just means that engineering and product/sales have to have a conversation, mutual trust, and a shared vision that extends beyond the next quarter. These are hard things to cultivate, however.
I agree. In whichever case, blaming individual engineers seems weird. Rarely are individual engineers to blame, and even when they are the organization should be robust enough to route around the occasional deficient engineer.
It's a full-company culture problem where ~everyone has personal responsibility.
Behavioral properties are end-to-end, like speed, security, customer success, uptime, etc. Everyone and everything on the hotpath for that has to be on board, as just one violator means nobody is achieving it. Operationally, part of achieving that means company norms, such as when collaborating, planning, executing, etc.
It's 100% fine not to provide any of these, and is even reasonable given it's hard to change one's DNA end-to-end... but better to be an explicit decision. Likewise, if starting a company... a lot easier to pick & ingrain these habits ahead of time.
I tried explaining this to some Azure product teams, and they gave me a blank stare in return.
Sure, you can have zone-redundant Azure SQL Databases, but not Azure App Service!
You can have zonal Azure App Service, but with a different network model than the default, so it's a breaking change for some apps.
It's as if nobody has actually sat down at Microsoft to build something similar to what their customers are building. It's all just tech demos, "get your blog hosted in 5 easy steps", and crap like that. Nobody at Microsoft has ever built anything of substance with the majority of their own platform.
It was the same story with Windows Presentation Foundation (WPF). It was hot garbage when it was first released, but Microsoft kept telling all their partners to prefer it over the legacy GDI+. People tried and failed to write GUIs in it. When after many years Microsoft finally tried to convert Visual Studio to WPF -- the first time they had used their own framework for one of their own apps -- magically the core issues were fixed and WPF become viable for real apps.
My experience with MSFT is that the account reps are clueless by design. Even the sales assistants with fancy technical titles are short of clues. Easier for them to oversell when they don’t know what’s really going on underneath. We have to go 3 or 4 deep into their product org to find someone who will fess up to product issues that we are already aware of.
Also they now require you to become Microsoft Partner, if you want to use the oauth2 login not only for personal Microsoft accounts but also other Microsoft accounts such as those provided by a business running Office 365. They changed it ~ 3 months ago and are now not able to verify people in time. Even if they just want to check your name, email and maybe a profile picture and nothing else. The process is much less obvious than for the same thing with Google or Facebook. Even the ZIP-Code has to include spaces as if it was written on a letter, otherwise you will not be able to send the form, aaaand there is no hint about this requirement.
In general, Microsoft, Google and others are behaving really enterprise-y in that they are slow, you cannot reach anybody of importance to solve massive issues with their products and everything is actually quite expensive for what it does.
I use Aurora just for this reason. RDS is just a great product. Doesn't even need the cloud sql proxy equivalent since you can just install the authenticator into the database.
NO, RDS includes non-Aurora versions of many DBs, plus MySQL and Postgres-compatible Aurora servers, plus MySQL and Postgres compatible Aurora Serverless.
Thanks for clarifying. I'd associated RDS with the non-Aurora offerings of Mysql and Postgres. Judging by upstream version compatibility it appears Aurora is more heavily forked than their non-Aurora siblings.
I don't think Aurora MySQL/Postgres are just forks, I think they are a completely custom datastore behind a MySQL or Postgres-compatible interface (which probably uses a lot of non-engine code from the open-source base database.)
Regardless it looks like anyone choosing Aurora should not hold their breath for Mysql 8 or Postgres 10+ compatibility. Seems like only one major version bump has happened since they launched the first one (Mysql 5.6-to-5.7).
Which is fine. It can just be a little confusing as they drift and the caveats grow.
> Regardless it looks like anyone choosing Aurora should not hold their breath for Mysql 8 or Postgres 10+ compatibility.
Current Aurora Postgres is compatible with pg 12.4; pg 10+ support has been around so long that several versions that support 10+ have already been EOL’d by Amazon. Even Serverless, which lags behind, is on 10.x.
WAIT. Be careful. That is a super expensive product with a high likelyhood of lockin. It doesn't support all SQL features. Also! I run hundreds of GCP databases and never ran into: "but the mandatory downtime on their Postgres CloudSQL", maybe it is only Postgres?
Same here. I have sent my GCP rep 5-10 follow up emails by now and still no response. It really feels like I have no one to talk to over there and I'm spending thousands per month.
That said, this is not some poor rep's problem. This is a problem of Google culture and one that will be very difficult to fix. They simple don't care much about their customers and never needed to.
I think you’ll find me surprising :) It’s surfacing and chasing down root cause of these misses that make things better for all customers, and that’s my goal :)
I helped a major user of GCP migrate off the platform to AWS for this exact reason. Totally insane that they still do this when AWS has had a rock solid offering in the form of RDS for like 10 years now.
Absolutely not. I'm sure it's happened but I have never personally seen an RDS instance go down, in over 10 years consulting and building in AWS. Google Cloud SQL was going down for multiple minutes every month for one of my clients, and their support just said there was nothing they could do about it. That cost Google cloud at least a million dollars in revenue over the next few years as this was a fast growing startup.
I’ve been a long time Azure user, their SQL Azure product was very glitchy and had constant outages (“just add retries!”), it’s gotten better in the last year or so, but it was totally unacceptable.
AFAIK that's a "feature", not a bug. They throttle concurrent requests and return a different error code compared to the on-premises SQL Server.
I just watched a load test hit that issue, the developers cranked up the traffic until the response times were measured in minutes. At that point Azure SQL started throwing errors related to too many concurrent queries.
Compared to just locking up or timing out, it's probably better.
E.g.: the 4-CPU pool has only 210 Max concurrent workers per pool, but if you think about it, that's over 50 queries running at once per CPU core, which is nuts!
Do y'all just like, ignore SLO's? Cloud SQL has a 99.95% SLO, going down for multiple minutes every month is within that. No smoke and mirrors here, there are ways to mitigate it but it's not Google messing around with expectations. HA doesn't mean 100% uptime.
Yet AWS manages to give near 100% uptime with RDS at a similar price point. CloudSQL downtime in the cases I've seen was caused by them rebooting both master instances at the same time. This is amateurish and totally unnecessary, the whole point of having multiple masters is to the ability to do maintenance reboots in a staggered schedule. This should be a trivial problem for Google to solve and would result in much better reliability for their users, yet it's been years with no change.
I had some instances that needed to be rebooted because they were running a vulnerable MySQL version. AWS gave six months warning (before forced reboot) and ended up extending it to 12.
When we first launched on GCP, there was no question that it was the way to go (frankly, because of BigQuery). Working with AWS, when we launched, was going to cost us significantly more up-front, before we had even brought on our first customer.
Fast forward 5 years... AWS has closed the gap in every way that matters. I still, frankly, trust Google's Security more than Amazon's, but I don't encourage folks to use GCP the way that I used to.
Just the opposite. In 2021, no questions asked, it's either AWS for general compute, or something more targeted if your business doesn't need it.
Until you get to the point where your bill is larger than a dozen engineering salaries, you won't get any respect from these people.
> I still, frankly, trust Google's Security more than Amazon's
If you have the time, could you expand on this? While I'm not directly involved in security at AWS, I'd be down to forward your thoughts to people who do.
I used to work at Google on Cloud, and am now an AWS customer. Have used both clouds extensively.
My comments are mostly backed up by my experience at startups and are not colored by my experience at Google (too different a beast).
GCP is great for teams that are also using GSuite because you can set permissions at the level of a Google Group and have them propagate to individual members. You can, of course, also create groups in AWS but they don't have the same semantics of Google Groups and don't cover the wide range of use cases that Google Groups does.
The AWS scopes -> policies -> roles -> resources chain of abstractions is less natural conceptually than GCP's GSuite accounts + service accounts with attached scopes per project.
Also the fact that each managed service (GCE, GKE, Cloud Builder) has its own service account that you can attach scopes to is really nice. GCP service accounts just feel more discoverable than AWS IAM roles - I think it's because the number of AWS pre-built roles is so overwhelming.
I think all of these replies so far capture my thinking. However, I think the simplicity of the GCP IAM model is what I will miss most going back to AWS.
I’m sure they exist, but over the last dozen or so years I’ve worked with public cloud offerings across 5 or 6 industries and domains, I haven’t found a use case that can’t be easily implemented in the simpler GCP model.
Gone deep on IAM in AWS, attempted a similar thing on GCP later for another project, was very surprised how weak GCP IAM was.
Top of mind things I found weak/weird:
- Basically can’t do least privilege, only want a role to read messages from one pubsub queue? Nope not possible
- IAM policies seem bolted on, legacy roles seem much better fit in the ecosystem, but they suck obvs.
- different gcp resources have somewhere between very coarse grain to just acceptable iam operations. Might just be GCS that lets you do proper least privileges policies like you would do in AWS.
I was very surprised how bad it was, AWS IAM is some black magic shit that is deeply impressive and often taken for granted, even GCP can’t replicate.
One thing that GCP is far better at is account setup. Having everything nested under a single gsuite organization with folders and projects and IAM flowing through is incredibly easy to work with and makes permissions simpler to understand. AWS has a long ways to go in this regard.
I came from an AWS background to my current company's GCP setup and was very confused at how IAM worked on GCP for a long time. Now that I know the system, though, I agree with you. It really makes a ton of sense and works really well.
The biggest problem I have with GCP is that something will say "you need the foo.bar.baz permission", and when I go to the IAM page to give that to myself... there is nothing in the search results for "foo", "bar", or "baz". Instead, I have to guess the "friendly name" for the permission.
I can totally relate. The amount of times I've spent scouring the docs for the "machine name" to put into TerraForm, or vice versa, to do through the UI...
I disagree. Once you learn IAM and able to segregate users into groups each with its own layer of security, then it is good enough.
Often the UI, and docs make it seem like everything is all over the place but AWS feels like lego with some pieces tucked away. That is where I think AWS can be improved upon with a better documentation UI and discoverability.
I do have to commend Google on Flutter + Firebase + Firebase Functions. I think if Amplify focused on serving Flutter users more it could pull me away from Google altogether.
Unfortunately, Google has done a fantastic job with making Flutter integrate with Firebase through Android Studio and there really is no product from AWS that matches its developer friendliness and low learning curve. This makes it very easy to switch.
I guess it is somewhat of a threat because the Firebase Cloud Functions also offer something of a counter to AWS Lambda as much as I love using it with API Gateway.
IAM shouldn't be a thing to learn. It's account management, default and easy to access options should be sane enough for most people to use. At big companies, sure someone has it as a dedicated part of their job description. But if you're in the majority of smaller companies, ones maybe that's just doing e-commerce and tech isn't their core skill set, account settings should be near invisible and still be trustworthy. It's not the Slacks of the world that have an issue with this, but the long tail of the world we now live in that software has eaten and companies are just scrambling to exist in it. Flutter integration is not in the list of concerns of this long tail.
And telling them to "just learn it" isn't the customer focused mindset, it's the engineering one.
I created all of the IAM roles that our 100+ person company uses. It was and is important from a security standpoint that we do not just blindly give too many permissions to employees. I had to do some research to understand what the bare minimum was and it wasn't too difficult to do.
Custom roles created through Terraform helped a lot.
It absolutely is required beyond small 3 person startups. Google is great to get started but when you are dealing with a dozen or more developers, especially at large organizations IAM offers that granular control and overview.
Yes its a bit of a pain having to add policies sometimes when you are first getting started but once it's up and running you can rest easy.
Learning it isn't that much more time consuming or difficult, its just a bit of effort that is all (we are talking a few hours at most).
What exactly are you disagreeing with? Segregating users into groups is possible with either platform.
The discussion is about GCP having much better functionality around it where projects and permissions are naturally integrated with gsuite organizations and users. This is objectively true. AWS has an archaic project system with a dozen different attempts at uniting it all but nothing comes close to GCP's smooth and easy manageability.
Thoughtful features like supporting UEFI Secure Boot with vTPM attestation. This allows building setups where even a full GCP account compromise can be mitigated.
Integration with our org GSuite (this alone is a massive plus).
> I still, frankly, trust Google's Security more than Amazon's, but I don't encourage folks to use GCP the way that I used to.
I trust virtually anyone's security over Google's. I've never had issues with AWS. I've consistently run into serious Google security failures. Google has airtight security for its own data, but not for its customers.
Examples range from Chromebook and Android security update policies (tons of expired machines on the public internet, in the case of Android, usually without people knowing), to pay-for-security on GSuite, to really difficult-to-audit Google Drive security (there's no convenient way to track and audit what was shared with whom or where data went), to just a ton of other things.
I've never seen Amazon be callous with my data. I've seen Google do things that even nineties "we don't need security" Microsoft wouldn't have imagined....
The only people I know who really trust Google security worked for or are close to people who worked at Google. There's a reality-distortion field based on how much Google invests in its own security that people fail to notice very basic failures, like millions of expired Android devices, or a lack of audit logs if someone physically accesses your machine to rifle through your gmail....
I had a similar experience with their recruiting. Some interviews were cancelled and had to be rescheduled only after I inquired.
Many weeks to generate offer and paper work and then 24 hrs to accept with a "take or leave it" ultimatum.
Just a very poor taste and I know some fine people who work at Google. Its unfortunate they don't seem to value warmth and humaneness when communicating externally
To be fair, Covid likely played a part in that long silence. I believe there was even a hiring freeze for a bit in spring. Not that it excuses bad behavior...
If the recruiter replied and said as much I would have been ok with that. At least they were communicating. Plus covid didn’t effect the US big time until March. That’s all of Jan + Feb they had to let me know something was going on.
They called me eventually were kind on the call. But the whole communication issues leading up to that really ruined it for me. I was initially quite excited too. I have no idea if this is common or if my recruiter was especially bad. Reading the parent comment made me think of the similarity.
Wow, that's a pretty horrible experience. Weird too since AFAIK recruiter/recruiting coordinatiors comp tied in some way to shepherding the successful candidates through the hiring process.
Me and a friend of mine interviewed in Nov and passed HC in Dec 2019 as well (different recruiters between the two of us). Our experience was different from yours: Immediate timely followups from the recruiting coordinators, team-matched and hired in the same month. Hopefully our experience is more common...
I know why. HR received so many resumes which they don't do any additional effort, do they receive compensation for each hire, no. Most of them are not FTE.
Sigh, I remember a similar experience. It was a third-party rep they pushed us to, but we would be asking for ways to re-architect one thing we already had setup on AWS, and all they would do is just try and upsell us on random offerings that clearly did not resolve our specific needs.
Some European-based customer apparently had a requirement if we engaged with them, that our service be offered via an acceptable vendor such as GCP, for some reason AWS apparently wasn't, but it was such a nightmare to even prod about an architecture that would have feature-parity with AWS, it wasn't even worth it. Also as an fyi, I'm no AWS fanboy, I don't use it in any of my own projects to avoid vendor lock-in this company suffered from.
Same terrible experience here with GCP sales and support, but the other options aren't much better. The reality is that unless you are in the 7 figure range, you don't get serious attention. I'm still surprised why sales is so dysfunctional but billions of quarterly profit means there's little need to change.
Since at least when we started spending about $100k/yr we've had a dedicated account rep we can contact at any time. They also get in touch to schedule a check-in every few months.
They've been genuinely helpful in several situations and have scheduled meetings with various teams around AWS (like, actual engineers) to get us answers to questions, support, and guidance. We've been put in touch with team leads and engineers working on beta features when we tried to use them and had issues to report.
Obviously this is all a sales tactic: if we have questions about X, putting us in touch with experts in X makes it more likely we'll successfully implement it and then pay them to use it. But it's the kind of sales we're getting value from, not just blindly pushing us to pay them more money.
This is for $100/MONTH!! That is the deal of the century.
And they are ridiculously helpful.
I don't understand this - AWS must be losing money at least on support side, though they obviously get happy customers (myself included).
And even at $150 this would be great.
I had a client on gsuite with google 8 years ago - we COULD NOT get anyone to help with some weird admin state flow issue - it just was not possible to talk to a human being for ANY amount of money.
I suspect that AWS is 'losing money' on this in the same way that Apple are 'losing money' on their high Street retail shops.
Working at a place with AWS enterprise support by contrast, for the second occasion, I would suggest that many of the places paying $15kpm don't cost that to support.
Yeah. AWS is probably "losing" tens of dollars per month hosting my personal account. They've made a few million dollars in sales as a result. I've personally started several projects on top of AWS which spend that much now. That started with the free tier back when AWS was young.
Google has treated me so badly so many times now on my personal account (as well as on business accounts, for that matter) in so many different ways that they've, conversely, lost MANY million dollars in business sales. It's hard to even count; a lot of people ask me for advise on decisions, and whenever someone even thinks about using Google Cloud in a business setting....
This is not a hole I see Google getting out of, except by eventually shutting down the Google cloud. Too many people have had too many bad experiences, and reputations take a long time to recover.
And the failures just keep on piling up.
Google is great for personal use, but I think they're diversifying in all the wrong directions. They're not structured for success there.
Google has been relying on it for years. It's completely seamless and means you get even better reliability by insulating from the underlying hardware issues.
Looking at support as a cost kind of misses the point. Bugs are bugs, and your big-ticket customers might just drop you rather than helping you figure out what is wrong with your product. Ignoring paying customers (even small-time ones) prevents you from improving your product. If they took the time to reach out (and especially if they're paying $100/month for support) you really need to listen to them and try and figure out how you can fix the issue so it won't affect big-ticket customers.
The price starts to go up steeply once you hit the % of monthly spend.
I'd guess they don't get many resource intensive support queries from the < $10k a month customers (and at that level you probably don't get the A team support)
Another comment later identified this - I actually did get a weirdly A-team support response. That's what made me scratch my head because I was coming in the poor / cheap / don't know what they are doing door (and most support is crap user misconfig issues at least from my experience). I just wanted the thing noted somewhere in case others were hitting it, instead I got a way above standard support specific technical response and some suggestions.
The other poster indicated it is possible for stuff like this that you get bumped, even on the el cheapo plan, to someone actually on product team. While I'd hate to be on the product team having to answer these, it perhaps keeps folks aware of customer issues so they can at least improve docs for corner cases?
That's what it says, but in practice I've asked some really general and technical questions of AWS support and always received a helpful reply without a paid support plan as well. With a paid plan the response time is better.
In general the AWS support has been great. In many cases, they've forwarded our requests to product teams who have even fixed bugs we've run into and contacted us directly.
Our other experience is with paid Azure support, which did little else than direct us to the (not related to the question) docs. They also had a really hard time understanding our technical questions about specific APIs. To their credit, they did eventually escalate to the PM of the service in question.
In general, the team responsible for the service really must be able to help out with support requests. In AWS this is definitely the case, in Azure as well but there's a bit of gatekeeping. Does developers and PMs in GCP participate in support?
Microsoft support is always useless in that way and the TAMs are pretty powerless. I hired an intern just to contest the hours to effectively cut our (large) premier bill 70-80%.
Their model was fault-based, and a “bug” gets billed to the support group. So the game was always for MS to avoid assignment for non Sev-A cases, and our game was to find a product defect for anything.
Or support helpfully suggests that you file a suggestion in the public Azure feedback forums.
Yeah... just like the other 5 dupes of the same suggestion with hundreds of upvotes that the product teams have dutifully ignored because it only matters to customers.
Google Cloud, AWS, MS Auzra are made to lock you into their system. I guess if fine if you are ok with it and you don't have an experienced systemadmin/dev op on hand.
There are plenty of cloud agnostic platform out there that does VPS, load balancers. digital ocean, linode, etc.
If you want to be green there is also the Advania they use geothermal energy. And since it is in iceland better data protection policy than US companies.
I'm not affiliated with any one of them, it is just from year of cloud provider hopping.
Couldn't agree more. I am on both DO and Linode for many years and I have had an amazing experience. Not affiliated with any of them, just a happy customer.
Have you tried deleting an S3 bucket? Go ahead, I'll wait. How about egressing any amount of data? I hope you have a Diamond Platinum AmEx attached to your account because you're about to lose your house with that bill.
Route 53, S3 storage, load balancer, everything is encouraged to work other AWS services! Smells much like what MS was doing. Don't get me wrong, it is more open than Google Cloud and Azura.
Somehow your comment reminded me how Google talks go at GDC and similar game development conferences.
While everyone else is talking about game engines, design and programming techniques, Google's talks are mostly around their cloud offerings, and customer telemetry.
The reason why the account managers change this way is because of so called territory or account realignment. These are sales terms but ultimately it means don't sit on one client for too long. Salesforce does this every single year, last week I received an email from 6th or 7th account manager since I started using them. The story is always the same: they get in touch,try to understand your business,ask questions,and see if they can spot something that previous guy missed. I hate it. However, this seems to be working on their end,as the sales keep growing...go figure.
So why is Google this way? They already have a terrible reputation for their non-existent support. Never thought they'd treat actual companies this way though. It makes no sense...
Inevitable. The entire company is built on a scalable business that required no direct contact with customers. Direct person to person interaction is the antithesis of their successful business model.
Of course, now when they enter a business that needs direct customer interaction, it shouldn't be surprising that they aren't good at it. No supportive processes and probably no internal appreciation for what reps do. Thus the turnover. They will have to build out a sub-org that nurtures and develops this part of the business.
I personally find customer interactions problems to be epidemic with fast-scaling internet businesses. It does however create a space for others to come in and do it better. We can hope...
Even their customer facing engineers who aren’t total a holes act like they are from a superior type of existence and we should be honored to be graced with their specialness without meaning to, but it is ground into them.
This seems to summarize my experience with software sales processes and interactions in general. I’m hopeful that software sales culture is beginning to evolve into something other than the pack of hyenas on a carcass that it is today. I personally have resorted to rejecting any and all conversations with software sales people unless I can dictate the direction of the conversation.
To be fair, BigQuery is probably their best product since Search so I can see why they try to milk it. It capitalises on what Google's best at, distributed systems engineering, while avoiding what Google's worst at, API design (by instead using SQL, which wasn't invented by Google). I say that as a customer who otherwise has little positive to say about Google.
As the guy who designed the architecture diagramming system, mostly to help folks keep straight what they're trying to help folks build, I hate that it's being used as a qualification/filter/etc. Sorry yo.
It's interesting that you've had that experience - it mirrors mine in dealing with Oracle reps: no engagement, no interest, very high turnover, always under pressure to sell, sell, sell. I wonder if appointing someone ex-Oracle to head GCP has carried that culture over.
You need to see GCP as its own brand with its own support team and policies (though they are impacted by some shared technical infrastructure and associated policies). I cannot say whether these are good or bad, only that they are distinct for GCP.
Experiences with support or lack of support concerning other Google product areas and divisions, especially those not designed for businesses really don't apply if you know how Google operates.
I don't use GCP in a personal capacity at this time and I work for a competitor (though am certainly not speaking on behalf of the competitor).
I use a lot of Google products. All of them I pay for, except search.
It is true there are individuals that shine, but across the board, Google sucks at support. It starts at the top with what kind of company they want to be, and goes from there.
Once upon a time I was one of 3 people globally that was the entirety of the GCP support team :)
You should have seen the extent to which the rest of the team and I tried to resolve customer issues.
Random App Engine app getting 500s calling an Open ID endpoint (the service wasn't strictly speaking managed by GCP). I absolutely could not reproduce this and had no other reports, but I knew it was happening for the customer. I examined everything that could be different... After 2 weeks I finally got it:
The customer app was running in a different data center compared to where I was testing. Calls to the API endpoint were timing out internally (pretty aggressive timeout interval). It turns out the Open ID service wasn't running in the same data center region. That should not have been the case but occurred because the internal BigTable service had to be upgraded in a particular region and hence all App Engine apps in that region were moved to another one - a fairly seamless process. The Open ID service unfortunately wasn't provisioned in that region (it's not a dependency of App Engine).
Once I identified the issue it was a matter of telling the on call SRE about it who then spun up Open ID in that region and thereby resolved the issues in a matter of 5 minutes.
Often a new support case meant discovering platform limitations - for example running into BigTable anti patterns because App Engine Datastore allocated IDs sequentially by default. That could cause poor read / write performance, in part due to automatic sharding behavior of BigTable. We later switched to snowflake IDs.
And then there were lots of Java folks used to Classpath scanning (default behavior of Spring Framework?) which for larger projects caused App Engine container startup to exceed the allowed time limit. That became a user education issue for which we then had to write an in depth article.
Good times :)
I absolutely hated dealing with billing inquiries though... so difficult to map someone's code to billable operations. Some SDK methods were of course utilizing multiple billable operations. Determining SLA violation refunds was much easier.
One thing that AWS Premium Support does very well is make situations like this seem unremarkable. This is the default way you do work. Going above and beyond is solving the issue, the next issue the customer hasn't asked you about, and giving practical guidance on things like cost optimization or availability concerns after weeks of deep diving.
I'm not saying you didn't do great work, but I think the secret to success at AWS Support is normalizing this level of customer obsession.
What I am saying is that this obsession was normal to us also. I didn't know any Googlers (whether in support or engineering) who didn't care about customers.
We did whatever it took to resolve a support case in the quickest yet most satisfactory manner, even if it meant doing some crazy research, meanwhile we got of course overloaded with more support cases coming in.
I had 100% customer satisfaction from all surveys. That was definitely hard to achieve because happy people usually don't complete surveys. :)
These were the early days of many GCP services. I worked there when Compute Engine and Big Query GA'd. This was just before and possibly during the time the paid support tiers were launched (I don't recall). I left mid 2013. No idea what their support is like now.
And those technical account managers just bounced customer questions to us 3-5 support engineers (technical solutions engineer)
Anyways, your AWS comparison may apply to the GCP support culture now (I have no idea) but back then we were all extremely customer obsessed - by choice.
I really appreciate that. Both as a customer, but also as a fellow practitioner.
At the end of the day, I don't think the complaints you hear about GCP Support / Service has anything to do with the people working on the line. There's always better or not better examples of folks you encounter as a customer.
As a support consumer, I've learned that my feelings about the person on the other side are very much colored by the fact that I almost never call support when things are going well or I'm in a good place. I typically call support when all else fails and I'm probably ready to throw a product out the window. That's not their fault, and it shouldn't be their problem, either, to a certain extent. They're humans, doin' a job, just like me.
As a support provider, I've learned that the same is true for all of my customers. I need to know that this isn't who they are (usually) when they go home to their family, or out with their friends. They're under a lot of stress and, one way or another, we've let them down and have to deal with that.
I think the challenge with GCP support is all about the overall strategy as a business. My experiences with GCP support is that I have to work way too hard to get past the robots and the scripted responses in order to find someone who is willing to listen and engage with my question or problem. And then, to make matters worse, about 80% of my interactions with Google support involves them trying to introduce me to some consulting company or third party provider. That is pretty much never what I want. I don't want some third party billing integrator that is going to mess up my reporting with their "value added" service. I already pay for support, I don't want to hear a pitch about paying for more support when I feel like I'm not getting what I've already paid for.
I don't hold this against the support folks at Google.
I hold this against the leadership at Google.
This is their strategy to contain costs or to juice sales.
If it were working, I think their numbers would be very different in the market. So the proof is in the pudding.
Something isn't working at GCP.
I don't think it's the engineers or the teams. It's the leadership and the strategy.
What you are describing unfortunately is due to the external vendor teams managing support -- a business decision as you rightly mentioned.
A lot of support became outsourced and the original support team became more of a tier 3 backline support with the first 2 tiers being handled by vendors. This started for GCP just after I left in mid 2013.
The vendors responsible for support have contracts to meet certain customer satisfaction metrics and time to first response metrics (based on customer tier and ticket priority). The majority of inbound support cases would get auto assigned to the vendor team. Only the top customers would reach Google employees directly.
The vendor and their staff simply have no incentive to troubleshoot an unusual issue - they also cannot have access to the tools that may expedite such troubleshooting. It's all about pattern recognition to quickly resolve a ticket. As many vendors were overeager to escalate support cases to the Googlers it became necessary to impose more requirements on the vendor team to be able to escalate an issue - invalid escalations were being tracked and the vendor would be penalized for the total of these.
I built the internal tool via which GSuite and GCP outages or significant issues would be managed from a support perspective. This had the advantage that vendors didn't need to waste their time (poorly) parroting (with a delay) the status of an outage as related incidents could be (manually) grouped together and then all be updated at once by the current incident manager on the support team - freeing up lots of support folks to work on other support cases.
I definitely agree this vendor-based support model is a frustrating experience for everyone involved. I witnessed that myself after leaving Google. By the way, I'm very surprised GCP tries to upsell you on consulting services based on your support interactions.
Running a support organization for a very technical product with so many unknowns and variables at scale is hard. I wonder who does that well and how they accomplish it (from an operational perspective).
I have no idea how AWS manages support internally. I work at Azure - until last week as a Developer Advocate, now as a software engineer in Azure CTO incubations - I have no idea how our support organization is run or what their quality is. However, from what I hear Azure support is eager to help anyone and everyone. I remember being very confused I got a phone call to inquire whether I needed help within a day or two of signing up for an Azure free trial.
Is paying extra 3-10% of your already expensive invoice worth the premium support that you _might_ need? Not sure I would consider that customer obsession... more like insurance.
Honestly given how good BigQuery is, I don't blame the sales rep for keeping their eye on their wallet. I know a lot of companies who are primarily AWS but use Google Cloud exclusively for BigQuery analysis.
(no, I'm not and have never been a Google employee)
I've used both GCP and AWS at multiple companies and personally, and my take is that a handful of important GCP products are significantly better than their AWS counterparts (Bigquery, Bigtable, Spanner, GKE), and a lot are just OK, usually slightly behind AWS in terms of features. If one of those products that's significantly better could be a differentiator for you, GCP is the better choice.
I'm not a rep, and I don't know anything about your use, but I'm 100% ready to learn about it and see if any of the stuff I know can be helpful to you and your team. Sorry this has been your ride to-date :( miles@sada.com
I'd say your experience isn't unique to google. I think at some point we are going to hit a place where people realize the convenience of the cloud doesn't outweigh the additional cost of having a mostly predatory business partner.
the convenience of the cloud doesn't outweigh the additional cost of having a mostly predatory business partner
Back in the dotcom days companies would spend a fortune on Sun kit but I bet when averaged out over time a comparable company would be spending a LOT more on cloud billing.
> Back in the dotcom days companies would spend a fortune on Sun kit but I bet when averaged out over time a comparable company would be spending a LOT more on cloud billing.
I would like to learn more about this. I'd have thought costs should go down over time. Are we doing more or is the cost per unit (not sure what that means) is truly going up?
I would like to learn more about this. I'd have thought costs should go down over time. Are we doing more or is the cost per unit (not sure what that means) is truly going up?
I think a number of factors add up over a 3-5 year timeline (obviously this will be more or less true for different organisations). There is the way the cost scales for a given instance in the cloud - in the old days, for example, doubling the memory or doubling the CPUs didn't double the cost of the kit, but it does for clouds VMs.
Another example is that the cloud bills you for everything, in the old days I could have a database server on a network and query it as much as I liked, the cost was fixed upfront for the lifetime of the hardware. Whereas it's very cheap to get started with a managed offering but e.g. BigQuery charges you for every query, Cloud Functions charge you per invocation, bandwidth is chargeable etc.
Speaking of hardware, in the old days you could look at your hardware and say, actually, it's fine, we don't need to upgrade/replace it this year after all, and it will just keep running. Whereas in the cloud the payment is continuous (perhaps offset by the fact that it's easier to "give back" excess capacity).
There will come a point at which the cost of DIY vs cloud will cross over, the question is whether you will reach that point, and if so, what you will do about it, since you may be well and truly locked in at that point.
When was the last time a landlord reduced your rent?
You always can drive cost concessions from sales, especially for base workloads where you have time flexibility.
For a big company, cloud rarely saves money for many categories of expense. In a normal market, it is almost always faster time to market to rent, and always cheaper TCO to own.
> When was the last time a landlord reduced your rent?
This is a different market and one which is ultimately constrained by the availability of land. Notably, cloud prices do fall especially relative to the compute power. Specifically I remember when Fargate moved to firecracker and prices fell by like 40% or something similarly considerable.
Maybe managing your own internal cloud is indeed cheaper (especially if you don't account for support or maintenance!), but arguing that cloud prices don't decrease or making some housing analogy seems like poor reasoning.
Maybe cars or trucks are a better example. The ROI of buying, leasing or renting a vehicle varies and the optimal answer depends on the scenario!
It’s always better for you as a person to rent box truck to move. If you’re a company that needs a truck 3-5 times a month, there’s a probability that leasing may make more sense.
I’d say that businesses that suck at managing on-prem will not magically get competent in a public cloud.
> I’d say that businesses that suck at managing on-prem will not magically get competent in a public cloud.
It takes time to build a competency. If you have to figure out how to operate everything from the hardware and networking all the way up to application, then you might very well fail as a business. If you can pay Amazon to manage your networking, hardware, databases, managed services, etc while you work on your application-level competencies, you stand a much better chance of succeeding ("I don't know if I could build everything from the ground up, but I can probably fumble my way to a container image"). As you become very proficient, you can cut costs, and if you're super proficient you might get off a public cloud altogether or more likely you'll just use that as leverage to negotiate a lower cloud bill.
Interestingly, you don't hear about many businesses that are so competent that they move from the public cloud to their own private clouds. Some people will shout "Lock in!", but I don't buy that--I've migrated between public cloud providers before and it's work, but if you're competent enough to run your own private data centers then you're plenty competent enough to migrate to them.
I haven't looked, so the number after the dollar sign might be going up or down.
But if it is not going down by more than inflation then the cost is basically going up.
My company has gone through 4 reps in 3 years. Every time we get a new GCP rep they just want to talk to us about "expanding our use of GCP offerings". The only thing they want to talk about is starting to use BigQuery - not my business at all.
I signed up for a Google Cloud Security summit, and afterwards a sales rep reached out me. It was obvious from the start they had no idea I was a gsuite or GCP customer. They then directed me to a NEW account manager (#4) even though I had been working with a different one. I had worked with the prior account rep going over our architecture to make sure everything was kosher (sustained use discounts etc). I even made them a schematic of our architecture on GCP at their request. Once I provided that to them I was met with radio silence.
It's really insulting, and to me obvious they don't care about my company at all. We're looking at other options.
I wouldn't be surprised if somehow this post leads to me getting an email from another rep "Wanting to start over and doing things right", which will inevitably devolve into the discussion of how can I use BigQuery.