I deleted my AWS account yesterday. It is obviously catered towards large organisations - very complicated tools and pricing that I couldn't really fit into my use case. I tried to just shut down the services that were using money but wasn't even sure I had found them all so I just closed the whole account.

I don't even like the idea of any of this stuff. I want to run my own little raspberry pi server or whatever, it seems much more fun and startupish than aws, which appears to be all of the corporate stuff I left (AIMs etc). This is funny because I remember AWS being thought of as great for "just experimenting with stuff".

It says here that closing your account isn't enough:

> Closing your account might not automatically terminate all your active resources. You might continue to incur charges for some of your active resources even after you close your account.

https://aws.amazon.com/premiumsupport/knowledge-center/termi...

Is that even legal? How is it the user fault that aws can't properly track and terminate the account resources? (Altough they seem very efficient in tracking usage when the goal is to charge credit cards).

They can do it but not instantly. This is similar to any other account, if you close your checking account but someone cashes one of your checks you're on the hook, if you return your rental car but there's highway fees they'll charge you, if you order food from Seamless but requested extra guacamole in a comment they'll add it to the initial bill, ...

Likewise here if your EC2 takes a little while to stop, or if S3 is a little behind in adding up your bandwidth usage, or if the billing for any other of the myriad of AWS products is not quite real-time, I don't see why that would be illegal. Expecting that an accurate statement can be presented within a few seconds of clicking "close account" and guaranteed to be exact just because they are "online services" is a little peculiar.

I agree it's reasonable in the circumstances you mentioned(as in, being charged for the resources until they are actually terminated, even if it might take a couple of minutes/hours for them to be terminated after closing the account), but the way I read it was like it could keep unterminated services running indefinitely.

AWS is great if you have a very large infrastructure budget. Playing around on a test AWS account where thousand dollars plus or minus is no big deal is a lot of fun. Playing around while trying to keep the bill under 10$ is just torture.

AWS has a free tier that lasts for a year. I actually used it to get hands-on experience with enough services that I was able to get my AWS Certified Solutions Architect, Cloud Practitioner and Certified Developer certifications. I easily spent 60 hours creating and deleting services and it didn't cost me a cent.

I don't even know how you'd accidentally get to thousands of dollars accidentally unless you were doing something more complicated like setting up an autoscaling group.

Spin up a t3a.micro EC2 instance and run it as much as you like - it's free. If you go into it (as I did many years ago) thinking, "Let me just spin up a 16 core server with 128 GB of RAM" because that's what you'd set up in your on-prem data center then of course you're going to have a big bill.

But that's not what cloud is about. It's about resources popping in and out of existence for the minimal time and resources needed to accomplish a task.

It's quite easy to burn more money than you want to. My startup had $15k in AWS credits and the other guy in the business set fire to it in no time, running windows based EC2 instances is a sure fire way to go through a lot more money than you realise.

Egress fees is one way to get a large bill on unrestricted static-file servers

Yep, the only way to do this is using aws-nuke [0], and the billing is such a mess for small use cases.

https://github.com/rebuy-de/aws-nuke

I'm really surprised that people immediately jump to cloud providers even for personal projects when you can lease VPS and server instances for much cheaper.

A buddy of mine runs a crypto currency validator in AWS and pays over $1600 a month.

You can rent a dedicated server with 32 cores, 256GB RAM and 2TB of NVMe storage for $400 a month. Tent two in different data centers for redundancy and you'll still paying half, not to mention no surprise monthly bills due to increased traffic or creating an expensive snapshot of your DB.

how is it still profitable after 1600 cost?

I shutdown the credit card i was using. They're intentionally making it hard and I decided it was less work to update payment methods on everything else.

Good point. I probably should replace all the credit cards I have with temporary ones...

Depends on your needs, scale and experience. If the product doesn't suit you, it's worthless (or very expensive depending on perspective).

If you just want 'a server', then AWS isn't the place to go to.

To elaborate a bit more: I tend to draw the line on IaC. For example: if you're doing GitOps with Terraform and Atlantis, then go for AWS. If not, you're probably not at the point where it makes sense anyway.

love digitalocean for this

In AWS, you can create everything under a Cloudformation stack, once you delete the stack, all its resources will be deleted automatically.

The problem however is all the beginner guides including those coming from AWS team themselves teach people to create resources via console without using a Cloudformation stack.

> It is obviously catered towards large organisations

And yet it seems that only the big cloud providers offer a reasonable authentication and authorization approach.

For many other providers, api keys lend full account access (no resource limitation, no read-only/subset of actions), which is unacceptable from a security pov.

Have you tried the billing explorer? It shows a pretty detailed report of what you use.

MFA needs to be forced on by default for all new accounts created with the UI. It is utterly irresponsible for AWS to not do this. People always counter with "you should know better!" But AWS markets itself to college kids. People were all up in arms and Robinhood allowing 18 year olds to create accounts which could result in unbounded losses, but AWS somehow gets a pass?

@aws, why not mandate MFA for a root user? in child org accounts where this is less feasible, you could allow access to the root user only from the parent account, no direct login at all.

One option is to set your root user's password to some random 64 character string and forget it. Any time you want root access (rare) you go through a reset flow, which means your root auth is tied to your email. Something like GMail has pretty strict controls so this is actually imo the safest option available.

While this may be safest, it doesn’t make sense why Amazon doesn’t save themselves a couple (hundred) grand in refunds by locking down root accounts.

You can keep the password unset as well.

You can create rootless organisation-managed child accounts. This is a solved problem.

Organization child accounts are not rootless, they're just seeded with a random password

Sure 'technically' you could call support and have them setup an email address for the root account and then it wouldn't be rootless anymore.

But if you setup your child account with something that cannot resolve as an email address it is no longer a working root account and won't be until you contact AWS support. You cannot change the email address setting yourself either, you cannot login and you cannot 'assume' root either (as if it were an IAM user). So in essence: no root access for anyone.

Then you add an SCP to deny support access from that account (or the entire OU or all org child accounts) and it can't self-contact AWS support either for good measure.

You just reminded me to set up MFA on my root account so thanks!

I migrated off AWS in December and closed my account. With the outages, they are likely looking at revenue shortfalls. Outages don't scare me, but their billing practices are already uncomfortable. This may make them prone to be more stingy about bill forgiveness. Bill forgiveness is inherent in the unsafe billing model which makes even having the account open even more of a huge liability.

I'm done.

Good reminder to completely close my AWS account. I have TOTP MFA on it but having a AWS account that had the same root login as my Amazon retail account was risky and a mistake from the beginning. Luckily I haven't used it for anything in years so it was as simple as following the "Close account" procedure.

I'll use Digital Ocean for anything small if I need to spin up a server in the future.

This happened to me. I had a 9k bill and I got hacked and stuck with it on my personal account. I gave AWS $100k business and that did not matter through being a decision maker at a startup I work at. AWS does not care about your business unless you are going to IPO. True story. Feel free to message me at Sgargconsulting --> GMAIL

This happened to me, the bill was so ridiculous that I wasnt even bothered by it. It got voided by aws support as predicted.

MFA does not prevent this. Its IAM keys.

This is an underrated point. Every "best security practices" guide you read has you setup MFA for console access, then create IAM keys with no such protection.

The credentials I'm using with Terraform require MFA in order to call 'AssumeAdmin'. Everyone I've ever shown this configuration has complained about it being overkill and tried to argue Terraform should just have IAM keys sitting on disk, one desktop compromise away from taking everything. And since it's used to provision basically everything it's a highly privileged account.

aws-vault works too

How big was the bill?

Like $120,000 after paying like $.30 for cloudfront for 10 months

It was pretty obviously uncharacteristic

If only they had some kind of charge limiting on accounts like their customers have been asking for years now.

I find it hard to believe AWS would try to suck blood from a stone. I’m willing to believe they get a 55k bill, but do they actually end up paying those?

I think you can get refunded for fraudulent charges, like if you get hacked. But if you leave an instance running or screw up auto-scaling somehow, that's normally when the charges may tend to stick around. I believe they have a one strike policy, though that could have changed.

No.

Thanks! You've reminded me to close my hobby AWS account. I will do that now.

Edit: Done! Was quite easy. Luckily I had no services running or needed.

their hardware MFA functionality is worse than useless

it only permits a single hardware token to be registered to an account

so good luck if you misplace or break your hardware token

When setting up the token you can scan the QR with multiple devices. E.g YubiKey and Authenticator App. This at least allows for a backup in case one goes missing. I agree it is kind of incredible that multiple tokens are not supported.

You actually can't mix hardware tokens and OTP apps. You're only option is to scan the code twice and skip hardware tokens entirely (which is quite reasonable, as the recovery for an app would be easier than for a failed/lost hardware token).

Note, though, that the new SSO login actually supports MFA in a normal way.

You create several users with admin privileges each with separate MFA.

Don't token limits reduce attack surface?

I did not like that limitation either and get around it by using the Yubico Authenticator app with three separate hardware tokens configured with the same shared TOTP seed. There's also path to reset your root account access, btw.

The point is to block unauthorized access. It is not less than useless. It absolutely does that. You really think if your lost your token Amazon would lock you out forever? Wrong assumption

Besides, just use AUTHY if that's your concern

FWIW, at a workplace I'm familiar with, the hardware MFA devices for a group 'disappeared' from the office during COVID and despite the company having an active relationship and contracts with AWS, it was taking over a year to get access to the root accounts in question reset. It is not an easy process.

That said, I have to imagine this is the wrong procedure and there's some way to duplicate hardware MFA devices to have redundancy for such a case...

That's bad and annoying but it is not at all worse than useless. A hardware token is a very worthwhile investment.

MFA is good advice

Also, I'd create a IAM user and severely limit your usage of the root account, and over time stop using the root completely.

Well, yeah, of course they're stuck with the bill. I feel like people think AWS is supposed to have infinite guard rails regardless of what the engineers using it do, like when people write code that infinite loops and it blows up their bill.

AWS gives money back in a lot of cases that I think they legitimately aren't responsible for.

I don't know that other cloud providers are going to do any better - an attacker who has your credentials and spins up 10's of thousands of dollars of infra will cost you thousands of dollars.

I'll certainly echo the advice for 2FA but, more importantly, use a strong, unique password.

I don't feel that some kind of quota management or predictable pricing is a significant ask. Most people or orgs do not need instant and infinite scaling. And would rather a 1 hour outage while they sort things out rather than a $100,000 bill for a minor bug.

Don't use a "cloud" vendor then? Just use a VPS vendor.

Should I just rewrite and host stuff like S3, RDS, ECS, Route53, Cloudwatch, Cloudfront, Lambda, etc. just because I want a spending limit?

Your argument makes no sense at all. You may feel like using production-ready hosted cloud services and still want a spending limit. Renting a single VPS might not solve all your issues.

Some people just want to use aws because it is convenient. I could just set up a vps with rabbitmq, nexus and postgres, setup cron to post systemd status and use the provider's cdn offering to accomplish most of that. Probably doesn't suit your requirements.

Why should a customer be stuck with the bill in the case of fraud? If someone fraudulently buys something in a store with my stolen credit card, I am not liable to pay for those purchases. Why would it be different for AWS services?

Right, but if someone buys say a shirt with your stolen card, it isn’t the store that picks up the bill. It’s the credit card company.

It depends on the merchant agreement, but merchants are often liable for the fraudulent charge, especially for online sales.

Sometimes the merchant does pay for a fraudulent transaction, and sometimes the issuing bank does. The credit card networks never pay. It depends on several factors including whether the card was physically present and whether it was swiped or used a chip.

This is just untrue though, my company has to eat the cost of fraudulent transactions and the burden is squarely on us to prevent it.

If you’re using Stripe or PayPal then yes it is the store owner who would be “picking up the bill”.

I'm surprised MFA isn't mandated by default. It is in Azure:

https://docs.microsoft.com/en-us/azure/active-directory/fund...

That link looks like MFA can be disabled but this chart shows otherwise (baseline versus opt-in enhanced): https://docs.microsoft.com/en-us/azure/active-directory/fund...

Yep, buying goodwill is very effective.

I know it's easy to get lazy about checking your AWS billing dashboard but I do it once a week - you can set up alerts and whatnot but I find it easier just to go look at the current usage to make sure nothing has gone awry.

I agree that you have to make it a daily or weekly habit to check because 30 days is too long a time to incur costs you're not aware of. That's why I built Billgist.com, same idea but it sends a daily email with usage amount right in the subject, like: $16.56 XYZAccount – Daily AWS billing alert.

I think the point is that someone could bankrupt the average hobbyist in an hour or two, if they were expected to pay it.

I think that rate of usage would trigger EC2 abuse alerts. Usually it's something that looks plausible but accumulated over 30 days.

The fact that AWS has no way to limit billing seems insane to me. Your only recourse for an accidental (or malicious) overcharge is beg customer support. It's an incredible liability.

They should either add technical measures to prevent overcharging on these types of accounts, or amend the terms of service to explicitly say that the account owner is not liable for charges incurred by an unauthorised user who obtains user credentials.

I find it hard to fathom that a company which sells machine learning as a service cannot detect that an account sitting dormant for months, which suddenly spins up several large VMs, has been compromised.

It's unsatisfactory that the only recourse is hoping for a favourable exercise of discretion on Amazon's part to waive the charges.

Jeff Bezos did not become a billionaire by being user-friendly. Guess anyone trying to cancel their Amazon Prime subscription would agree.

>Guess anyone trying to cancel their Amazon Prime subscription would agree.

The unsubscribe button was easy enough to find, but had to maybe go through 2 nag screens before letting me cancel. I guess it's not pro-user, but it's also not exactly a roadblock to me canceling.

Well, it's not a roadblock but it's a dark-pattern and user hostile. Just like the lack of spending limits on AWS.

Compare that to the experience of unsubscribing from netflix, for instance.

> Well, it's not a roadblock but it's a dark-pattern and user hostile. Just like the lack of spending limits on AWS.

Yeah I explicitly say it wasn't pro user. I was more disputing the "Jeff Bezos did not become a billionaire" (presumably by making it hard to cancel prime) part.

Well, I didn't really mean to say that what made him a billionaire was making it hard to cancel amazon prime, that was just an example I came up with that demonstrates Amazon general stance on user-friendliness.

Still, I'm not sure I understand what you're trying to dispute. If we agree with both premises(1. Bezos is a billionaire and 2. Amazon products aren't usually user-friendly), than I maintain my position that the expression "Jeff Bezos did not become a billionaire by being user-friendly" evaluates to true.

fair enough.

If you were "hacked" (ie password reuse), the attacker can just relimit this.

There's still a lot they could do around that though, e.g. requiring that billing details be re-entered upon change or sending an email alerting the user with instructions to secure the account if it was not authorized.

Just as likely as basic credential compromise is lateral attacks on compute resources from vulnerabilities such as Log4J.

Enabling MFA, restricting intra/inter-VPC access, removing hard-coding credentials from configuration files/source etc., switching to SSO/removing user accounts with passwords, creating and applying restricted IAM roles, and applying those reduced privileges to EC2/ECS/EKS instances are all things that and should be done as soon as possible. (Non-exhaustive, but illustrative list)

AWS could simply require the user verify they have access to the card or other payment method on file.

That is super impractical when you run 200 AWS accounts in production.

The option to set a limit doesn’t require you to set one. Assure lets users have a limit without any significant issues.

Does AWS require distinct billing accounts for each project? I use GCP, I can attach the same account to multiple projects. (It also lacks this basic billing limit feature, unfortunately.)

No, that's why it would be silly to have to change it for every account. (and also problematic for a team with a bunch of child accounts to then have to gain access to the org's card or something like that to change settings for their childs)

I don't follow. But they could make the quota feature a toggle (that has the same mechanic to unlock). Easy, easy enough for a junior dev. And I'm sure folks spending more than a few minutes chatting about it could come up with an even better solution.

This is the reason I refuse to use AWS for anything that I work on in my own time. I don't want to wake up to a $20k bill because someone thought it would be funny to DDoS my weekend project.

Limiting billing is ambiguous. The computation spent is spent, unless AWS can predict how much you are going to cost by even initiating certain operation, which is not going to be straightforward for all the services they provided.

It's not hard to invert the transaction, ask the billing service if an action can be done instead of bill upon completion. Done. This isn't that hard. If your credits won't make it to the end of the month, send out the email warnings to top up - which should be predictable unless there's a large spike hitting the account. If you prefer invoice billing because you never want your services going down, it's trivial to accommodate with this system design.

Sure, big warnings about data loss/services down in the event of zeroing out your credits.

Having such a safety net would really help the thousands of people who sign up for transient reasons. I don't recommend a noob set up an AWS account. I also don't let my kids play on the freeway.

In theory, all IaaS services could be built like an Ethereum node:

1. fill an account with abstract "multi-resource usage credits" (or gas);

2. have each API call specify how many credits the caller is willing to spend from their account on the operation (a gas limit);

3. operations in progress are tracked using real-time resource accounting (= distributed tracing, but cheap-as-possible at the expense of granularity);

4. any operation that goes over-budget is cancelled (even if it completed successfully, the user doesn't get the benefit of that completion), with the user being charged exactly the limit they specified.

(Yes, that means that the IaaS eats the overhead costs of any additional processing they did before noticing and stopping the workload going beyond its limit. This incentivizes the IaaS to optimize their scheduling, at all levels of the stack, to minimize the amount of "overhang" work it allows.)

And yes, to be clear, this would "infect" every layer of every service. If the IaaS provided a DBaaS, the DB's query planning and execution layers would need to know about these limits and accounting mechanisms. Etc.

I'm not saying it's practical. But it's definitely possible, with no foreknowledge of cost.

azure seems to manage that just fine with their free azure credit that they give to MSDN subscribers ($150/month spending limit, refilled every month, no chance of going over).

They bill by the request or minute for most of the services. Billing for storage is a little more complicated, but many services could easily be limited.

Somehow Oracle of all companies is able to get this right with their free tiers for their cloud services.

This does not stop the high bills from your account being hacked. Just reduces the amount before you notice. spinngin up gpu and ec2 resources until limits are hit in how many are created is an operation that takes seconds.

Can be stopped with Budget Actions.

AFAIK setting a budget or alerts does not say "Not authorized to spend more than X amount, deny any spending over this limit". It will alert you, yes. (Also, not on by default)

Can you set a "hard" budget? Like, never let me get more than $20/mo in services from AWS and after that just disable my account?

No, that's not possible.

And these can then be turned off by the hacker.

You can execute Budget Actions to stop services and revoke permissions. It’s not simple but it’s possible.

Services like Lightsail have caps and flat fees. AWS needs more of this.

please, everyone enable MFA and billing alerts in your accounts. do not commit access&secret keys to github.