Thanks, I have looked at this as well. There are a few issues:
- It only scales worker nodes
- It only scales nodes if a pod can't be scheduled. This is too late for us as are requirements are to scale when resource utilization crosses a threshold (ie reserved memory is over 70%).
- It still doesn't solve the baked AMI problem. I may be wrong about this but the AWS docs for CA are lacking. For instance, when does the ASG/launch config get created? Is that something it does or it needs to be done ahead of time? Which AMI does it use? etc.
1) Yes, but I'm not sure why you would want to scale other nodes (I assume you're referring to the masters?)
2) Yes, the cluster-autoscaler will give you more node capacity. If you want to scale Pods on a CPU basis, you can use a Horizontal Pod Autoscaler (https://kubernetes.io/docs/tasks/run-application/horizontal-...). I believe it now support custom metrics as well, so you can scale on any resource threshold.
3) The ASG/Launch Config is created by Kops when you create the cluster; and is also manageable through the kops tools. It will default to the kops default AMI (which includes the kubelet, and everything you need for k8s to run), but you can also override that with a different AMI if you have a k8s compatible AMI that more fits your needs.
When used with Kops, it should give you what you're looking for (with the baked in kops AMI)