You'll probably want something closer to a GAN like pix2pix - https://phillipi.github.io/pix2pix/

An example implementation would look something like edges2cats https://affinelayer.com/pixsrv/