This looks pretty interesting, for a few reasons: it can fit into existing ray tracing rendering pipelines, and you get most of the ray tracing benefits (reflection, shadows from geometry, refraction, depth of field, camera geometry) along with it. These are both pretty big.
Render quality is high / equivalent to MipNeRF (or however it’s capitalized). PSNR is equivalent or better, and the rendered output can be denoised with, say OptiX.
Some downsides/caveats — it works best if you retrain a little, so you won’t get the best quality if you’re pulling over mipnerf trained Gaussians, it’s slower to render than a straight rasterizer, like 50% slower, and of course these splats still don’t have geometry to them, as is much discussed elsewhere.
They spent a lot of work optimizing doing this for Nvidia’s RTX series, and the raytracing task is a little different than the typical one, which is to say it’s rare in ‘normal’ raytracing that you’re adding up the colors of 100s of transmissive, semi-transparent colors/radiances to get a single pixel; usually the bulk of the color from a raytraced scene comes from a smaller number of rays. If this method becomes popular, then NVIDIA could no doubt optimize the raytracing architectures further in the future and you’d get back some of that speed.
All this to say, I hope this gets rolled into existing engines, it’s practical engineering that would add a lot of options to workflows, and pretty neat!
I hope with this we can stop using "splats" to refer to the primitives that are being trained and rendered. The primitive are the 3D gaussians, "splat" was always an implementation detail of the rasterization approach.
The anonymity on the web page may be due to requirements of the venue where the paper is currently under review. Sometimes there are rules against advertising preprints under review. Or maybe they have a link to the web page in the submission and want to keep it double blind this way.
I don’t think this offers anything new on relighting — this is literally just adding the ability to sample a splat during a raytracing render step; nothing about changing a splat’s color / illuminance / whatever. That said, you could turn on, say, reflections for parts of a splat scene subject to whatever rules you wanted.
Maybe you could get relighting here of a sort, by computing some sort of normal to each individual splat as you hit it, and checking light values. That said, the splats don’t (as far as I know) approximate geometry, so normals on them may not be (won’t be?) what you’d expect from actual geometry.
I’d say this would be worth a try, it might turn out to work okay, and would be nearly free in the rendering pipeline they describe.
Render quality is high / equivalent to MipNeRF (or however it’s capitalized). PSNR is equivalent or better, and the rendered output can be denoised with, say OptiX.
Some downsides/caveats — it works best if you retrain a little, so you won’t get the best quality if you’re pulling over mipnerf trained Gaussians, it’s slower to render than a straight rasterizer, like 50% slower, and of course these splats still don’t have geometry to them, as is much discussed elsewhere.
They spent a lot of work optimizing doing this for Nvidia’s RTX series, and the raytracing task is a little different than the typical one, which is to say it’s rare in ‘normal’ raytracing that you’re adding up the colors of 100s of transmissive, semi-transparent colors/radiances to get a single pixel; usually the bulk of the color from a raytraced scene comes from a smaller number of rays. If this method becomes popular, then NVIDIA could no doubt optimize the raytracing architectures further in the future and you’d get back some of that speed.
All this to say, I hope this gets rolled into existing engines, it’s practical engineering that would add a lot of options to workflows, and pretty neat!