I am familiar with solutions like Depot, but the problem with them is that they are also building on a remote host. We already have many powerful VMs setup for building. The only inefficiency is that they are not sharing cache layers. I could probably hack something together with rsync and such, but would ideally have a more robust solution.

Docker's distribution (formerly registry) with a redis cache, buildx using the cache-(from¦to) directives. There's another flag I'm forgetting that signals the builder to ship all layers.