They do write CUDA code, oh boy do they ever. PyTorch is just a coordinator for CUDA or sometimes Metal kernels. New AI architectures and algorithms often end up needing a new or tuned kernel. Look at Flash Attention for an example of one of those that had a big impact.