Definitely not. My company has been drastically scaling back our AWS spend and I mean about a $10m+ a month reduction. I guarantee we're not alone. All the gee-whiz resume-driven-development type project infrastructure was the first to go lol.
It's easy to accidentally waste a lot of money on AWS. My company cut our costs from ~20k/month to ~6.5k/month with no degradation of UX or dev velocity. It was all due to unnecessary redundant builds and integration tests and lingering unused resources. We used the CloudForecast service to identify and track costs (https://cloudforecast.io/). An underrated side effect of this service is that making costs visible to the engineers incurring the costs can encourage them to spend less.
I’ve seen a big bank spin up a dev environment in a cloud for a lending platform at a cost north of $36M a year before they finished the requirements for the project.
Not impossible at all to believe they could cut $10M/month from could spend at a big enough company just by trying.
I guess the question is why would you wait to make savings like that? $10m/month would pay for a small 'cloud cost cutting' team with plenty of change left over.
seriously, that's a very beefy server per engineer, just in cloud resources. not including what they use to do their day to day work. NFW would I allocate such a spends per engineer--maybe for beefy projects/groups, but not for the org as a whole.
It’s not just costs incurred by engineers, it’s costs incurred by users. If you’re providing file storage as part of your app, then for every engineer with a dev environment in the cloud, 1000 users could be uploading gigabyte files into your S3 bucket.
I work for a company with ~1000 engineers, and our build server spend is much lower than what you suggest. If we were spending even $10k/mo on build servers, heads would roll.
$1M/mo will rent you about 300 c5d.24xlarge servers, at on-demand rates (and of course no medium or large enterprise is paying the higher on-demand rates). I'm pretty sure we're running somewhere on the order of 10 build servers, and they're maybe 4xlarge or 8xlarge boxes.
I know of a few companies burning AWS at around that rate, though they are outliers. And yes, 90+% of that is lighting money on fire. In the list of business priorities, cost containment is pretty low for some companies when capital is cheap. There is a much larger cohort of companies spending much less but wasting just as much.
I once saw a major west coast health insurer purchase $50MM+ in hardware to solve what could have been at most a couple million software problem. In fact, had they negotiated well they might have been able to pawn the whole problem off on their vendor.
And here I was pretty proud of myself for reducing our monthly bill from $5K a month to $2K… we have some basic sports data APIs that are hardly being used since there's no live sport being played, so I've removed most of our redundancy for the time being.
A $10m/month reduction is quite significant. Question is, was your company spending $10m/month extra unnecessarily or have they had to cut down and go without important things in getting this kind of reduction?
A reduction in customer numbers means a reduction in the resources required.
Many businesses associated with outdoor activities, sports, travel etc etc will be seeing less traffic at the moment. Many business are pretty much closed. This will then severely impact other businesses which sell to those businesses, and so on, all the way up the chain to advertising and cloud providers.
The Ubers and Lyft's save by committing to a certain level of spend. They have paid -- might as well use the resources, anyways -- unless they've since negotiated with Amazon.
It seems likely to me that a recession or pandemic affects more than just the OP's business, so while that one customer is probably meaningless to AWS, the thought process that led to scaling back spending could be meaningful.
0.3% of monthly revenue from a single customer's reduced spending only a couple months in to a multi-year recession is pretty alarming.
Makes me wonder if OP's company is particularly exposed to retail, travel, or another sector severely impacted by the first order effects of the virus.
That sounds like a lot to me for just one customer. But in the absence of much more data about AWS customers and revenue, it's hard to draw any meaningful conclusions.
My team reprioritized efficiency over features during the recession, trimming a large amount of unnecessary AWS spend that wasn’t as pressing when the economy was performing well. That specific spend will never come back to AWS, because it was always unnecessary.
It’s probably not for the entire year, a lot of it is just various clusters being scaled back. That said some of the stuff has been moved to colo hardware, and some of it is probably just gone forever.
Maybe, maybe not. But I think I agree with you- 10m/m is a comically large number(I wonder how many companies spend even 1/100th of that a month on cloud compute?) and I have no reason to believe this person so why would I?
Correct there is no reason to believe me. However for context and perspective our quarterly revenue is usually around $3 billion. $10 million is not comically large. I’m not talking about a startup. hth
I'm shocked that some middle manager with half a technical head on their shoulders didn't use this as a VP play before this; surely, "I saved the company 120m a year with no downstream consequences to yada yada" would have been worth it for someone somewhere to climb the ladder.
Why would you, if in good times you can enlarge your own budget/headcount/importance/... just by increasing budget no questions asked, that's a safer way ahead. You risk devaluing your own position in the company for an uncertain gain...
I was obviously being slightly sarcastic. I wouldn't want to personally work that way. I do believe that might be quite an effective strategy if one's goal were climbint the corporate ladder.
I think it would be easy to hit especially if you were spending on cloud SAAS offerings + actually have a lot of users. I know I've managed to spend a few $10K/month on a project before - and it was actually celebrated because it was less than half the cost of what it was replacing.
The cloud is the opposite of recession proof. Why do you use the cloud in the first place? Flexibility, scale, ease of operations. What are you gonna do with your servers during a recession, put them on Craigslist? Are you gonna put your entire datacenter on there too?
The same mechanic that makes the cloud attractive in the first place is what will bring the hammer down in a serious recession. Companies will turn every stone to use less capacity and save money, and they will easily be able to do it.
Every industry is going to face / implement a degree of cost cutting, that's unavoidable. However the control and scalability of cloud within an ethos of cost cutting is attractive for this exact reason. A business can enter that market and know they have a very flexible amount of control over their expenditure. You need to keep in mind its only a minatory of companies who have migrated to cloud, the majority have plans, but had never made the crossover as yet. With necessity being the mother of all invention, this adoption will only accelerate now
Well, the cloud is supposedly 'scalable'. And by that we all thought about 'scaling up'. But scaling down or shrinking is 'scalable' too and now is the time to find out if that's going to happen. Ironically, more businesses right now need to be scalable in terms of shrinking without dying than the original premise of being scalable (growing bigger and bigger without interruptions).
On-prem servers don't run themselves: there's hardware maintenance, depreciation, security, networking, license renewals etc etc.
The typical customer I see who's moving from on-prem to cloud has a clear trigger/deadline in the form of hardware reaching EOL or licenses being up for renewal, and these aren't going away with COVID.
On prem equipment can be depreciated, and you can drag out the useful life of the equipment you own if necessary. Cloud opex evaporates each month, never to be seen again.
CTOs/CIOs call the shots when times are good, CFOs when times get lean.
There's still the aspect of trading higher total cost for lower initial investment when you use a cloud provider. That won't change and there might be more incentive to think short term.
the power bill is only one small component of the cost of physical servers.
The main cost is the up-front capital. This capital has a cost (aka, the cost of capital in finance parlance), which either come from debt, or financed by equity, or in opportunity cost.
Therefore, shutting the servers down will not decrease that cost of capital, and so continue to cost the business that cannot make use of the servers in a recession.
The great thing about cloud servers is that it's pay per use - so there's no cost of capital. And in bad times, the expense can be eliminated, and directed else where that's more needed.
Right now a more important question is how fast is the Cloud growing? Will it outgrowth the recession?
The Cloud is still growing as many businesses are still moving towards it. If the natural growth of the Cloud is bigger than how much the economy shrinks then the growth will still be positive or flat.
I work for a tech company in this situation. It has grown a lot the past years and was expected a lot of growth in 2020. Now the expectations are more close to a flat growth with heavy losses in some quarters and growth in others. It is a similar case to the Cannabis industry. It may shrink from the reduction in economy, but still grow to the end of the year as it is just starting its growth path.
I think growth of cloud business - net / net - will decrease in 2020.
1) The basket of customers who may do a big Cloud migration may shy away from big CAPEX projects (unless it's a very low hanging fruit that saves money).
2) It's not completely obvious whether cloud saves money, it really depends on the customer case by case.
3) The big sales pitch for cloud migration was so it can add features faster. Adding new features isn't on top of people's minds right now
4) Existing cloud customer will find ways to optimize their cloud spend.
Maybe. But that's assuming people will use more cloud services at home than they and their companies do when working at full capacity.
I was thinking that we were going to start having blackouts due to everyone being at home during the day running their A/C, until I realized that countless businesses are now not running their A/C since no one is in the office/restaurant/mall.
In what recession do people not retreat to their homes? Recessions are usually associated with higher levels of unemployment, presumably with that extra time spent at the home (as opposed to going out, and spending money).
Stay at home is a short term change in the behavior. But people losing their jobs and changing their spending habits will have long term effects as well.
For the big 3 cloud providers, even if their cloud did contract significantly, they are all poised to earn more on the other parts of their business to make up for it.
Microsoft is heavily invested in software that helps with remote work like office 360 and Microsoft teams.
Google is getting more traffic than ever I'm sure with people stuck at home and looking to buy or do things online. All of which are advertising their services in Google.
Google doesn’t make money from increased traffic, they make money in ad spend. It doesn’t matter if more people are staying at home if the bottom falls out on the ad buyers.
Somewhat compensated by other ad venues moving online. I don't think all the signs and billboards in Times Square will be as popular as they were before.
The cloud appears to be a short term gain to achieve operational readiness and efficiency, but it becomes a long term loss in terms of ongoing operational costs.
AWS bandwidth costs, even with their volume pricing tiers, are >10x as much as you'd pay than if you'd bought it directly.
On a small enough scale, it doesn't matter so much. When you're paying millions a month, it matters a huge deal.
Even on a micro scale:
* Vultr, DigitalOcean are 2-4x cheaper than AWS (and they are established companies with a track record of reliability, as well as 20+ locations around the world)
* Even AWS offers "Lightsail" which are significantly discounted VPSes with included bandwidth that costs more than the Lightsail price! Their ToS prevents you from using it as proxies... lol.
* Want to save money on S3? Get some colo space, buy some Enterprise HDDs and set it up on RAID, and your opex is going to be 5-10% the cost of S3. Or SSDs if your access patterns require it. Set up some SMART-based email alerts and an offsite backup procedure, and it's absolutely reliable enough for the vast majority of business use cases. P.S. even S3 silently loses files...
* Using CloudFront? It might be easy and integrated, but it's absolutely not going to be the cheapest or best value. It's really marked up, heavily. Spend a few hours talking to leading competitors and you'll get a quote for half your CloudFront bill, if not less; often for a better CDN.
* Trying AWS Rekognition, or other ML-as-a-service products? Hire some data science intern and they can copy/paste the leading open source project from GitHub, train it on your data, and you'll quite possibly get better results than generic ML solutions... for as low as 1/100th the cost. (Real example).
Yes, you gotta maintain it, but I find that people generally think maintenance is a bigger monster than it actually is.
AWS has huge profit margins: where do you think it comes from? Hint: it makes it from you.
If you’re spending $100k per year on cloud services, then in 10 years, you’ll have spent $1 million. How many software licenses and hardware can you buy for that?
Granted, some services are only online, so you’re at the mercy of your provider on what they want to charge.
What everyone is missing here is that cloud is a secular growth trend. It's not that it's recession proof but rather that it will be growing less fast in a down economy but still growing.
It depends on how many of the cloud providers customers are venture backed unprofitable startups as opposed to major corporations. I suspect that there are relatively few major corporations that would trust GCP, many who would trust Azure and AWS.
I wonder with contracts like that whether the short term gains from having people tied in to contracts also has some longer term negative consequences?
Fortunately I've not had to work with salesforce before, but if Id just signed a big contract with them on the expectation of a growing business and then they were inflexible and enforced the contract to the letter while the rest of the business suffered then it'd leave a bit of a sour taste. Having said that, I guess if you've got your shit together you'd write clauses in to the contract to protect you from stuff like that e.g. early termination with a bit of a fee.
I remember a blog post, I think from Tim Bray, about 15 years ago or so arguing that the people in IT who had great careers went into government during recessions or down periods and out of government during boom times to work on the things that they had specified during the down period.
So maybe Government is recession proof. Seems about right.
My own opinion is that "The Cloud" will fail only if things get _really_ bad. Many who are currently laid off may be getting to that point; but for the cloud to actually fail it has to be more than just some customers. It's got to be businesses and power users.
Given how power users are likely to be "Essential", and how businesses might scale back but otherwise avoid major changes, those demographics will only pull out past a different inflection point. It will take things being dire on a wide scale; whole companies and industries failing. When that happens society as a whole can be seen as failing.
If we do ever get to the point that "cloud fails" we should really hope for a new new deal.
It's pretty easy to cut down clusters to the minimum number of nodes for them to function (say cutting 10 servers to 2), or downgrade the size of instances to compensate for lower traffic. There's no need to restructure infrastructure in most cases, if anything using a cloud provider makes this easier.
It's "easy" to do this, but it's also just as "easy" for your service to fall over when you do it wrong. Capacity planning and performance engineering is a lost art in a world where you would simply throw money at the problem until it went away.
It's not that hard (I've done this recently for clients). If your traffic is 90% of normal for a few months, scaling down with load makes a lot of sense, then you can scale back up when ready. There are also autoscaling solutions like serverless.
This is true for simple services. It's different when you're dealing with something like a Cassandra or Elasticsearch cluster, where the system isn't stateless and scaling up takes some time for re-balancing.
Sure, it won't apply to all services, particularly datastores are best left alone, but if you need a quite few web workers say which sit in front of your search or db to service millions of users, and can cut that in half during low traffic times, you may save significant money, as the user pram above has noted.
GP was saying there is no flexibility of infrastructure if you use cloud servers and they are therefore recession proof.
I was saying it's pretty flexible within the constraints set up, and often companies have spare capacity they could significantly scale down to save money (for example a travel company during a pandemic has very few customers), so it is flexible in the most important way (cost).
That means cloud companies will see a significant reduction in resource usage and thus revenue if enough customers do this.
The short version is lockin. Unless you get very careful ops engineers making it truly cloud-agnostic, you can't even move from AWS to GCP without a huge amount of overhead.
I'm a contractor and several of the companies I've worked with recently have almost hard-coded their services to expect certain things available to them on GCP or AWS that aren't as easily available on the other.
You can argue that they should use 12facter and real DevOps and portability and so on but that's just not what people are actually doing so they've locked themselves in and for companies that are struggling to pay their hosting bill already, saying "We need 3 months of 6 developers' time to move" isn't going to go down well.
This is why we choose to sit everything on top of OpenShift, we get Red Hat support 24/7 to run it and if we needed to migrate to a different provider its far less painful as OpenShift abtracts the cloud providers API's away from us. The only real changes we need to make our admin based. Technical changes are very minimal.
OpenShift costs quite a bit. Have you compared, costs- and ROI-wise, the OpenShift approach with an alternative of having a small team of consultants (or internal team, if available) architect your platform in a cloud-agnostic way, based on vanilla K8s, plus required additional OSS packages?
No matter how careful you are to avoid lock-in, infrastructure has weight. You have data, network infrastructure, security audits, IAM permissions, Infrastructure as code (no Terraform is not a magic bullet), training, DNS entries etc.
I wonder how many people who think they have a large “cloud agnostic” infrastructure have actually sat down with project managers and come up with a project plan and cost analysis of a large migration.
I have witnessed how much it costs to do just a Workday migration.
If you've invested the effort and personnel in being able to move from some cloud provider to some other cluster/cloud solution, it probably already makes sense to just get your own servers and move yourself there, since you're basically capable of doing that (and your costs will be cut by a large factor).
Different industries are affected by recession in different ways. Generally speaking, recession proof industries are affected much less than the general economy. Some examples of recession-proof industries include grocery stores, alcohol manufacturers, gas stations (funny in today's reality), some entertainment segments (movie making and music business), national defense. Generally speaking, if it's a necessity (food) - it's considered to be much more recession proof than discretionary spending industries. IT or cloud are two general terms but parts of them can definitely grow during recession.
I wouldn't count on even supposed essentials like food being all that resistant. It used to be that people couldn't cut their food spending significantly because it was mostly the basics of bread or rice to survive. Those days are gone across much of the world. I could cut my food spending to a fairly small fraction of what I am now if I truly needed to.
There's a big difference between what you CAN and what you WILL cut. Alcohol and tobacco aren't necessary to survive. But alcohol consumption is some countries have actually gone UP during the lock down. Not sure about cigarettes, didn't see any stats. We'll see what happens to the food industry. Milk consumption is WAY down, because up to 50% of milk products were sold through restaurants. But wheat and rice producers may not be negatively affected or may even gain on the recession. Think about this in the following terms. Yes, you can cut your food BUDGET by 30% and survive. But you won't be able to cut your CALORY intake by 30% without starving.
Just to put out a reference here because I have seen numbers in German news yesterday. Alcohol sales for wine and beer are up around 20-30%. In grocery stores. But is that really a hint at overall alcohol sales growing?
For me, I know I buy way more alcohol now in grocery stores. But what about all the alcohol not spent in bars and restaurants??
it's about as recession proof as it is idiot proof. As long as people are around, I'm sure businesses will bounce back and overtime, things will begin to return to normal.
The "cloud" has always held a fluid definition. If you ask someone "what's the cloud?" you'll get different answers depending on who they are. So, just being a generic term makes it to give this a clear "yes" or "no" answer.
What should be clear is that anything denoted by "cloud" quite literally comes down to leveraging computing power provided by someone else in one arrangement or another.
Anyone providing or relying on a "cloud-based" service is directly part of a marketplace and it's innate dynamics which includes recessions and the complex fashion in which those play out.
While it's highly unlikely to see Amazon shutter AWS overnight because demand implodes entirely, I wouldn't be surprised to see changes in their pricing or offering to offset macro-economic effects in the long term.
The other side of the spectrum would be specific niches disappearing. For instance, small companies that leverage cloud infrastructure and resell hosting as an offering to small businesses. Many of them might find themselves out of business if their customer portfolio consisted of recession sensitive businesses as well.
