BTW, your parenthetical 3rd paragraph is a fairly deep question and fast approximations to it would be useful beyond just rendering scenes realistically. I assume you're familiar with the use of lenses to implement FTs long before they were computationally tractable (https://en.wikipedia.org/wiki/Fourier_optics)?
The PINN idea seems interesting but it may or may not work out; if you're interested in that kind of thing, there's more of it in Dercuano, Derctuo, and Dernocua. Some of the ideas in there have been tested and work, others have been tested and found not to work, but most are still untested.
As for Fourier optics, yeah, I worked with a guy last millennium who had worked on some optical analog pattern recognition stuff at university. I've never gotten it to work myself; I suck at lab work so far.
BTW, your parenthetical 3rd paragraph is a fairly deep question and fast approximations to it would be useful beyond just rendering scenes realistically. I assume you're familiar with the use of lenses to implement FTs long before they were computationally tractable (https://en.wikipedia.org/wiki/Fourier_optics)?