> My recommendation is to treat CF as a single point of failure. Once it gets in a broken state, you may have to destroy your stack and rebuild
One of the more common scenarios where CF gets into a broken state is:
1) create new S3 bucket + something else (e.g. some Elastic Beanstalk env update)
2) something else fails, causing rollback
3) S3 bucket already contained data (e.g. your failed Elastic Beanstalk env update caused it to write data)
4) CF refuses to destroy the S3 bucket, entering a "rollback failed" state
In this cause, manually wiping the S3 bucket works well enough. But generally, it appears that CF works kind of when the updates you're making are really small, incremental updates.
Sometimes it gets totally corrupted and you need to nuke stuff, per your advice. This automatically leads me to the following suggestion: leave mission-critical data out of CloudFormation. Specifically, stuff like RDS databases which you absolutely never ever want to have destroyed: just provide the endpoint as an input to your CF template.
You can set the DeletionPolicy attribute to "Retain" to work around this S3 issue. CloudFormation will successfully rollback without attempting to remove the S3 bucket. You can then do so manually yourself before trying to deploy it again.
You better do so before deploying again, because the roll forward will break since the resource already exists.
This a major pitfall when using DeletionPolicy=Retain with named resources. It breaks seamless rollbacks/rollforwards. If you rollback, in order to deploy again you need to either delete the named resources with DeletionPolicy=retain that were rolled back, or update your template to rename them all. It is such a huge pain.
True, but it beats the alternative where CloudFormation deletes objects that you didn't want deleted. The underlying issue is that the S3 objects are outside of the CloudFormation scope, thus it takes no risk and doesn't delete your objects.
A nice feature would be a "ForceDelete" deletionpolicy where it would delete the objects. You can even set this initially when creating a stack, and change it to "Retain" later when the stack is stable.
Totally agree btw that it's a huge pain initially, though once you know it it's also not that hard to work around.
My preferred behavior would be for CFN to not barf when rolling forward. In other words, to be able to assume control over a resource that already exists.
Specifically, stuff like RDS databases which you absolutely never ever want to have destroyed: just provide the endpoint as an input to your CF template.
There are ways around it as stated below, but I agree completely. I don’t bother with CloudFormation with cross shared, building block type infrastructure like RDS and ElasticSearch. It’s just not that much of a pain to spin up a database on each account. Besides, the characteristics of the databases in different environments are going to be so different, that you are going to either have parameters or FindInMap functions anyway so for all intents and purposes, you’re not running the same template anyway.
As the article said, changing any resource that’s exported from a CF template is such a pain, it would be better just to use parameter store if you can get away with it.
Try to avoid naming resources ("Name" property), so there would be no clashes. Use Ref (same stack) or ImportValue (other stack) to reference created resources. If you want, you may concatenate (Fn:Join "AWS:StackName", "-public-elb").
Cloud way is you replace things, rather than keeping it. It is convenient to know, that if I delete CF stack, everything is cleaned.
Speaking of S3, its better not to include bucket resource in CF until you know what you're doing.
> stuff like RDS databases which you absolutely never ever want to have destroyed
That begs for a separate CF stack, with template creating RDS and related things only, then export endpoint in Outputs.
One of the more common scenarios where CF gets into a broken state is:
1) create new S3 bucket + something else (e.g. some Elastic Beanstalk env update)
2) something else fails, causing rollback
3) S3 bucket already contained data (e.g. your failed Elastic Beanstalk env update caused it to write data)
4) CF refuses to destroy the S3 bucket, entering a "rollback failed" state
In this cause, manually wiping the S3 bucket works well enough. But generally, it appears that CF works kind of when the updates you're making are really small, incremental updates.
Sometimes it gets totally corrupted and you need to nuke stuff, per your advice. This automatically leads me to the following suggestion: leave mission-critical data out of CloudFormation. Specifically, stuff like RDS databases which you absolutely never ever want to have destroyed: just provide the endpoint as an input to your CF template.