Docker stopped publishing builds of their own registry many years ago. So if you want to run the official registry, you need to build it from source. This leads to a fun and exciting bootstrapping process; to launch the registry, you have to pull it from somewhere. Since your registry, which is where you'd like to store it, isn't running, you can't pull it from there. So you have to use some third-party registry to bootstrap. Or do what I did, and give up, and just watch their registry crash randomly when it receives input that confuses it.
People will make fun of me if I go into the great details of the workarounds I have to make a DigitalOcean managed Kubernetes instance pull images from a registry that's hosted in the cluster. But it's fun, so here we go. I use a DigitalOcean load balancer to get HTTP traffic into my cluster. (This is because the IP addresses of Kubernetes nodes aren't stable on DigitalOcean, so there is really no way to convince the average browser to direct traffic to a node with any predictable SLA.) I configured the load balancer to use the PROXY protocol to inform my HTTP proxy of the user's IP address. (I don't use HTTP load balancing because I want HTTP/2 and I manage my own letsencrypt certificates with cert-manager, which is not possible with their HTTP load balancer. So I have to terminate TLS inside my cluster.) Of course, the load balancer does not apply the PROXY protocol when the request comes from inside the cluster (but the connection does come from the load balancer's IP). Obviously you don't really want internal traffic going out to the load balancer, but registry images contain the DNS name of the registry in their own names. The solution, of course, is split-horizon DNS. (They should seriously consider renaming "split-horizon DNS" to "production outage DNS", FWIW.) That is all very easy to set up with coredns. It is easy to make registry.jrock.us resolve to A 104.248.110.88 outside of the cluster, and to make it resolve to CNAME registry.docker-registry.svc.cluster.local. inside the cluster. But! Of course Kubernetes does not use cluster DNS for container pulls, it uses node DNS. Since I am on managed Kubernetes, I cannot control what DNS server the node uses. So the DNS lookup for my registry has to go through public DNS. I created an additional load balancer, for $5/month, that is only for the registry. That does not have the PROXY protocol enabled, so when someone DoS's my registry, I have no way of knowing who's doing it. But at least I can "docker push" to that DNS name and my cluster can pull from it. This is all fine and nice until you make a rookie mistake, like building a custom image of your front proxy and storing it inside your own registry. What happens when DigitalOcean shuts off every node in your cluster simultaneously? Eventually the nodes come back on and want to start containers. But your frontend proxy's image is stored in the registry, and to pull something from your registry, it has to be running. This results in your cluster serving no traffic for several hours until you happen to notice what happened and fix it. (I do have monitoring for this stuff, but I don't look at it often enough.)
And that's why I have 99.375% availability over the lifetime of my personal cluster. And why smart people do not self-host their own docker registry.
> building a custom image of your front proxy and storing it inside your own registry
But do the images have to be co-located with their registry?
Can't the images be somewhere else, and the registry replicated among the nodes, so any node can find its image through the registry and fetch from that location?
Docker stopped publishing builds of their own registry many years ago. So if you want to run the official registry, you need to build it from source. This leads to a fun and exciting bootstrapping process; to launch the registry, you have to pull it from somewhere. Since your registry, which is where you'd like to store it, isn't running, you can't pull it from there. So you have to use some third-party registry to bootstrap. Or do what I did, and give up, and just watch their registry crash randomly when it receives input that confuses it.
People will make fun of me if I go into the great details of the workarounds I have to make a DigitalOcean managed Kubernetes instance pull images from a registry that's hosted in the cluster. But it's fun, so here we go. I use a DigitalOcean load balancer to get HTTP traffic into my cluster. (This is because the IP addresses of Kubernetes nodes aren't stable on DigitalOcean, so there is really no way to convince the average browser to direct traffic to a node with any predictable SLA.) I configured the load balancer to use the PROXY protocol to inform my HTTP proxy of the user's IP address. (I don't use HTTP load balancing because I want HTTP/2 and I manage my own letsencrypt certificates with cert-manager, which is not possible with their HTTP load balancer. So I have to terminate TLS inside my cluster.) Of course, the load balancer does not apply the PROXY protocol when the request comes from inside the cluster (but the connection does come from the load balancer's IP). Obviously you don't really want internal traffic going out to the load balancer, but registry images contain the DNS name of the registry in their own names. The solution, of course, is split-horizon DNS. (They should seriously consider renaming "split-horizon DNS" to "production outage DNS", FWIW.) That is all very easy to set up with coredns. It is easy to make registry.jrock.us resolve to A 104.248.110.88 outside of the cluster, and to make it resolve to CNAME registry.docker-registry.svc.cluster.local. inside the cluster. But! Of course Kubernetes does not use cluster DNS for container pulls, it uses node DNS. Since I am on managed Kubernetes, I cannot control what DNS server the node uses. So the DNS lookup for my registry has to go through public DNS. I created an additional load balancer, for $5/month, that is only for the registry. That does not have the PROXY protocol enabled, so when someone DoS's my registry, I have no way of knowing who's doing it. But at least I can "docker push" to that DNS name and my cluster can pull from it. This is all fine and nice until you make a rookie mistake, like building a custom image of your front proxy and storing it inside your own registry. What happens when DigitalOcean shuts off every node in your cluster simultaneously? Eventually the nodes come back on and want to start containers. But your frontend proxy's image is stored in the registry, and to pull something from your registry, it has to be running. This results in your cluster serving no traffic for several hours until you happen to notice what happened and fix it. (I do have monitoring for this stuff, but I don't look at it often enough.)
And that's why I have 99.375% availability over the lifetime of my personal cluster. And why smart people do not self-host their own docker registry.