After 7 years, and me wanting to move off EKS since I got the job 4 years ago, we are moving to ECS (I rose to power of Lead recently, but my engineers also thought it was a great move as they're sick of all the K8s BS).
The time sink required for the care and feeding just isn't worth it. I pretty much have to dedicate one engineer about 50% of the year to keeping the dang thing updated.
The folks who set it all up did a poor job. And it has been a mess to clean up. Not for lack of trying, but for lack of those same people being able to refine their work, getting pulled into the new hotness and letting the clusters rot.
Idk your workload, but mine is not even suited for K8s... The app doesn't like to scale. And if the leader node gets terminated from a scale down, or an EC2 fails, processing stops while the leader is reelected. Hopefully not another node that is going down in a few seconds... Most of the app teams stopped trying to scale their app up and down because of this ...
I would run on ECS if AWS was my cloud at a start up. Then if scaling was getting too crazy, move to EKS.
But for the love of God ... Keep your monitoring and logging separated from your apps. Give it its own ECS cluster, or buy a fully managed solution. It is hard to record downtime if your monitoring goes down during your K8s upgrade.
The time sink required for the care and feeding just isn't worth it. I pretty much have to dedicate one engineer about 50% of the year to keeping the dang thing updated.
The folks who set it all up did a poor job. And it has been a mess to clean up. Not for lack of trying, but for lack of those same people being able to refine their work, getting pulled into the new hotness and letting the clusters rot.
Idk your workload, but mine is not even suited for K8s... The app doesn't like to scale. And if the leader node gets terminated from a scale down, or an EC2 fails, processing stops while the leader is reelected. Hopefully not another node that is going down in a few seconds... Most of the app teams stopped trying to scale their app up and down because of this ...
I would run on ECS if AWS was my cloud at a start up. Then if scaling was getting too crazy, move to EKS.
But for the love of God ... Keep your monitoring and logging separated from your apps. Give it its own ECS cluster, or buy a fully managed solution. It is hard to record downtime if your monitoring goes down during your K8s upgrade.