Seems like what we need is a specialized compression algorithm. Since the domain is largely constrained it sure seems like there's some low-hanging fruit here.
Lots of tradeoffs. If you invent a new codec, it's unlikely to make it into hardware for a while (even AV1 encoders are not yet as widely supported) and therefore you will have to do encoding and even decoding in CPU, which takes away resources from the workload. h.264 is still probably the best general purpose codec for real-time desktop streaming - low bandwidth requirements, 444 support, build to lossless, low latency, moderate CPU usage if GPU is not available, and well supported in GPUs for a long time (e.g. even back to Kepler).
