shorter wavelength light really doesn't like existing. It takes more energy to produce, there are way fewer possible materials to make mirrors out of etc. Just look at how much trouble the industry had doing EUV lithography.

Even UV wavelengths aren't terribly small, and the shorter the wavelength, the more energy it has and the more likely it is to destroy whatever material your optical CPU is made of.

Not sure what gives you that idea. It seems unlikely that there are materials that can withstand billions of x-ray pulses per second and continue to function without being altered. They might exist, but the higher the energy to get to low wavelengths for fast and information dense computing, the increasingly implausible it gets that a suitable material is physically possible.