I saw Greg Nichols give a talk about this at the UICC [0] this year. It was a very entertaining talk covering the business and technical aspects of this project.
From the start the art team on Big Hero 6 wanted to make a really big movie. They envisioned those big sweeping shots of San Fransokyo that made the technical folks shudder. They have a massive rendering farm at their disposal but still probably wouldn't be able to render all the shots in time for the release; for one reason, because the art folks kept making changes, and for another, because they contain so many entities. A handful of folks on the rendering team built a proof-of-concept renderer with support for the global-illumination look they wanted while also being able to handle the massive scale.
As the linked article mentions, the solution was to optimize a massively-parallel algorithm by finding coherent rays that can be efficiently calculated together. It's a batch method: start casting a bunch of rays, identify similar rays and group them, calculate collisions for each group, cast reflected rays, identify similar rays and group them, etc. What I like about it conceptually is that in a way it treats light as a field rather than as individual directed rays.
Their initial tests were rendering an infinite plane of generated buildings. The dystopian metropolis was mesmerizing, and I think Disney would be wise to come up with some dark plot line to set there. Anyway, someone way at the top (the director or art director I think) saw the proof-of-concept renderings and decided that it was the perfect tool to attain their vision of this movie, which, by the way, has a hard release date something like three months away.
So they proceeded to turn the proof-of-concept into a production-grade renderer in time to render all the frames and save the movie. And somehow they pulled it off, they released a really beautiful movie, and they wrote a paper about it, too.
Hyperion is an amazing renderer. Normal renderers are limited by memory, but as Hyperion traces millions of rays at the same time they can exploit coherence between rays and they can load in and subdivide geometry on demand. It also allows them to efficiently handle PTex textures, which the folks at Disney really like.
Here's the paper that explains some of the renderers details:
I'm going to deflate the marketing spin a little bit since large companies tend to go down the route of 'there was an impossible task and we use outside the box thinking to do what no one else could'.
Tracing batch rays, sorting them for better coherency and sorting the shading for coherency are not new ideas, but they are long over due to be used heavily in offline, cpu, ray traced rendering situations.
Large amounts of geometry have been done in movies for a long time, so Disney didn't really do anything that couldn't have been done before. All the same tracing a path of light one at a time while interlacing the shading is terrible for performance so it is great that Disney was able to reap the benefits. It is something that should have been obvious to anyone who understood cache performance in CPUs, but now at least people have something non-academic to point to.
Given that this is HN, I'd encourage reading and commenting on the substance of the technical paper rather than the marketing webpage.
Those ideas have indeed been around for a long time; the paper nicely cites all of the related research work. (Including among many others, a SIGGRAPH paper I wrote on the topic 18 years ago.)
I think the paper is excellent. First, there's a big gap between academic papers on a topic and the experience of actually building a real system that works for real movies. It's unusual for people in industry to take the time to write up their experiences building these systems, so I salute their making this contribution to general knowledge about rendering systems.
It's easy to carp about "all could have been done before"; that seems like an argument that could be applied to try to dismiss just about anything.
I'll just say that I don't want to dismiss anything except for exaggerated marketing. I think it is reasonable to say that I should ignore it though.
> I think the paper is excellent. First, there's a big gap between academic papers on a topic and the experience of actually building a real system that works for real movies. It's unusual for people in industry to take the time to write up their experiences building these systems, so I salute their making this contribution to general knowledge about rendering systems.
I'm %100 with you. I love seeing the results of cache coherency used in a real scenario. I was only trying to say that the actual imagery could have been done in other renderers, albeit with more pain and I would guess more attention to level of detail.
So I'm not trying to downplay anything except for the idea that the actual movie itself couldn't have been done without cache coherent batch ray tracing, when in reality it is a big optimization that came from a bit of a leap of faith, which I think is significant enough to pay attention to.
I completely agree with you that the information in that paper is excellent and I think it's amazing that Disney is so willing to share so much information about their renderer, even though their sister company Pixar sells a renderer.
The question though is whether or not this paper should have been published at EGSR. Personally I think that an EGSR paper should either propose a novel idea or should provide a good survey of the field. I don't think this paper succeeds at either of those. Their method is 'simply' a combination of already published ideas and I don't think that they do a very good job at comparing those existing ideas.
Personally, I think this paper would have been better suited as a talk at SIGGRAPH, a publication at JCGT or a technical report (like Pixar). So: great paper, wrong venue.
I'm actually really glad they got the paper out there as a small pdf (and I'm actually glad they compared it to other commercial renderers). The reason is that it shows very real world performance results. Not only that but cache coherency optimization results are shocking and illuminating to many people because it seems so counter intuitive when there technically aren't more instructions being ran.
Every paper I read now I am looking for all the things that were left out, how the comparisons have been changed for each scene to make that particular algorithm look good etc.
It answered a big question lingering in my mind that I haven't been able to actually try out.
From an engineering standpoint I think that Hyperion is an amazing renderer and I'd love to dig into its source code. From an academic standpoint however I think that their paper on it wasn't very interesting. I don't mind reading about the details of a production renderer, as there is a whole lot of info on writing production renderers. But I don't think that it should have been published at EGSR. There were no real new ideas in that paper that were worth publishing. The paper could have had some merit if they provided a good comparison between their architecture and other architectures, but sadly they decided to compare their renderer with other off the shelf renderers. Very little is known about the architectures of those renderers and as they all have their own implementations of acceleration structures, texturing and other shared components, they aren't comparable at all. It would have been much nicer if they implemented their algorithm in a framework such as Mitsuba, so that it could be compared with other algorithms.
I do think you should give them more credit on the engineering side though. Using large amounts of geometry has been done before, but renderers such as RenderMan and Arnold require that all geometry fits in the memory at once. Tracing large batches of rays at the same time allows you to load in or subdivide geometry on demand, which makes the size of the harddrive and the size of the individual meshes the limit, instead of the memory being the limit. Their scheme also allows for creating good ray packets, which clearly is needed if we want to reap the benefits of Moore's law, as those extra transistors mostly translate to wider SIMD vectors instead of higher clockspeeds and simply increasing the amount of children of a BVH node won't scale well. Reordering the rays allows you to create coherent packets out of incoherent rays and it allows you to efficiently handle motion blur, as all rays in a packet need to be in the same time interval, which allows you to efficiently make use of those wider SIMD vectors.
I'm certain they could have produced Big Hero 6 without Hyperion, but Hyperion gave their artists a lot of freedom by removing some of the technical limitations.
> There were no real new ideas in that paper that were worth publishing.
The real world speed improvements from specific methods are the significant part. There are many ideas floating around, not all of them work out once you discover all the comparisons and data omitted from an academic paper.
>I do think you should give them more credit on the engineering side though. Using large amounts of geometry has been done before, but renderers such as RenderMan and Arnold require that all geometry fits in the memory at once.
This is out of core tracing and is great that they implemented it, but neither out of core tracing and sorting rays are feats of engineering.
It has actually been done many times in both academics and commercial renderers from at least a decade ago. SIMD is also used to various extents in any decent renderer to varying degrees of success. Sorting into packets isn't the only way to use SIMD, or even the only way to use to scale SIMD use to wider lanes. Interestingly Skylake should have fast gather/scatter operations which will change the effectiveness of various techniques.
Basically good design choices and some practical knowledge from trial and error has been heavily exaggerated by marketing. I would guess that the actual render programmers would say the same. This isn't a breakthrough, it is a refinement and that's good enough. I love what they did, but to carry the torch that the impossible was made possible is disingenuous.
I say that based on what I've seen and claims I've heard - optix, prman unloading and loading procedurals, a renderer making that claim a long time ago (maybe finalrender?) In any event I don't know if any work particularly well, for the same reasons that Hyperion succeeds of course.
I'm actually more surprised that you would think of sorting and batching of rays as a feat of engineering, I can't imagine it seems all that difficult to you after all your experience.
> As the linked article mentions, the solution was to optimize a massively-parallel algorithm by finding coherent rays that can be efficiently calculated together. It's a batch method: start casting a bunch of rays, identify similar rays and group them, calculate collisions for each group, cast reflected rays, identify similar rays and group them, etc. What I like about it conceptually is that in a way it treats light as a field rather than as individual directed rays.
How does this differ from "coherent ray tracing" and "ray packets", like people have been doing the last 15 years?
They're generating millions of rays to trace them at the same time, so they should be able to use packet tracing and they seem to use that [1]. Incoherent rays can be extremely troublesome when tracing packets, as you could be entering a branch of the BVH for just a few rays in the packets and in that case you're wasting a large part of the SIMD vector. There has been some research to alleviate that [2, 3] and creating more cohorent batches alleviates it even more. One of the other big problems of packet tracing is that there is no good way to handle motion blur, but if they're generating millions of rays at once they can just create packets with rays that are all in the same time interval.
I've already gone beyond my understanding of the technical details. The paper probably goes into it (linked in the sibling), and it also looks like it cites Wald.
Just noticed it also has the proof-of-concept city scene I mentioned on the last page.
What's the difference between path tracing, ray tracing and photon mapping?
I'd never heard of path tracing before now and the video makes it sound the same as ray tracing/casting, but what I've read implies otherwise (but I'm not sure how). Clearly it's different to photon mapping, but what ever happened to that: Was it too computationally expensive for the pay-off?
Ray tracing is a more generic term that covers rendering techniques which trace rays between a camera and a scene. Back in the day everyone used Whitted style ray tracers to render shiny metal blobs because secular reflections are trivially handled with ray tracing. Unfortunately other physical aspects of light like diffuse reflections were not handled by early ray tracing algorithms.
Kajiya introduced the rendering equation, a mathematical formulation of global illumination (takes diffuse, secular into account). It's a multidimensional integral equation. Unidirectional path tracing is an algorithm he introduced to solve the rendering equation. It uses several rays traced from pixels in the camera bouncing randomly through the scene. It's a Monte Carlo integration algorithm for solving the nasty integrals.
Photon mapping involves a pass algorithm tracing light from light sources into the scene, then a pass (like path tracing) gathering that light back at the camera. It better handles more complex light effects like caustics.
> It better handles more complex light effects like caustics.
I wouldn't say better. I would say it provides an approximation to the rendering equation more quickly than path tracing does. However, photon mapping is a biased algorithm, which means that if you average many independent renderings together, they won't converge on the correct (exact) image. Path tracing methods (bi-directional, Metropolis, etc.) converge on the exact solution, regardless of how noisy each individual rendering is. (However, it may be the case that an unwieldy number of samples is required for tricky caustics, so in practice, path tracing may fail to produce a correct result because of high variance.)
Photon Mapping might be biased, but it's extremely easy to make it consistent by using the method outlined by Knaus and Zwicker [1]. Using that method photon mapping will converge to the right result. Even without progressive photon mapping you can choose a photon radius that won't cause visible errors.
Ray tracing is the art of tracing rays and detecting collision.
Ray tracing is used in photon mapping and path tracing.
With photon mapping you trace a ray from a light source and store the light energy for the area where the light hits an object.
With path tracing you ray trace a ray from the camera and every time you hit an object you check if you can see a light source from that spot. If true you return all the gathered energy back to the camera (pixel).
The other answers here are correct, but to shed a little more light on it, the main development of path tracing over ray tracing is that path tracing involves Monte Carlo integration to simulate many effects, the big one being Global Illumination, but depth of field and soft shadows as well. What this means is that at every step along a ray path, you carefully pick a new direction for the next ray segment that is randomized according to whatever effect that segment corresponds to, whether it's the surface material there, or the camera lens or the area/volume light source, or something else. The implication of this randomizing / Monte Carlo integration is that it typically takes many ray paths to get the color for a single pixel. (You can use only one or just a few, and your result will be very noisy.)
When people talk about ray tracing as something different than path tracing, they usually mean the first form of ray tracing that only did direct diffuse lighting, point light sources, simplistic materials, and multiple bounces of perfect specular reflections. (Aka recursive ray tracing.)
Note that not everyone makes a distinction between path tracing and ray tracing, you can easily find lots of examples of people referring to ray tracing and talking about GI and Monte Carlo techniques. "Ray tracing" is evolving to incorporate modern developments.
The way I understand it; ray tracing and path tracing are the same conceptually except for how so called 'global illumination' (GI) is calculated. Ray tracing uses a short cut that computes the value with an equation that can be applied efficiently to the entire image. Path tracing on the other hand calculates the GI by tracing not only direct rays from the camera lens bouncing off an object to the light sources but also follows indirect rays that bounce around multiple objects in the scene on their path from the camera to the light source.
Practically speaking path tracing is going to give you a more real to life image at the expense of a slower rendering process.
This is wrong. Ray tracing simply is a way to determine visibility between two points, just like the Z-buffer. Ray tracing won't produce a color on its own. Path tracing on the other hand is a method which uses ray tracing to solve the light transport integral. It creates paths between the camera and the light using ray tracing and then it calculates the contribution of that path to the image. Most toy ray tracers only handle specular reflection and transmission, and direct illumination. This is called Whitted ray tracing and is probably what you're referring to.
You're probably right, but nobody I know calls it "Whitted ray tracing" unless Turner is there in the room, or some point is being made about or in reference to his paper specifically. ;) In my experience, graphics people generally do mean producing colors for pixels when they say "ray tracing". But I think physicists would assume "ray tracing" means visibility queries and nothing more.
When someone says 'ray tracing' to me today, before they elaborate on what they mean, I assume they're talking about using straight line segment visibility tests probably to make some kind of picture, probably using Monte Carlo techniques, probably for global illumination. So, they might mean path tracing, or one of the (many) other ways to simulate lighting, or my assumption might be wrong, but that's okay because language is fun, and vague, overridden and imperfect terms allow us to have longer conversations. :)
Is the light transport integral isomorphic to what Heckbert's paper [1] refers to as "gratuitous partial differential equations [2]," as explained by Fuller [3]?
>The “begetted” eightness as the system-limit number of the nuclear uniqueness of self-regenerative symmetrical growth may well account for the fundamental octave of unique interpermutative integer effects identified as plus one, plus two, plus three, plus four, as the interpermuted effects of the integers one, two, three, and four, respectively; and as minus four, minus three, minus two, minus one, characterizing the integers five, six, seven, and eight, respectively [3].
The Hyperion renderer was made by Disney, not Pixar by the way. Pixar only uses their own RenderMan renderer. Disney uses a mix of Hyperion and RenderMan, as Hyperion is not useable for test renders, as it can take up to 15 minutes until a pixel will appear on the screen, as they trace millions of rays at the same time.
"Disney uses a mix of Hyperion and RenderMan, as Hyperion is not useable for test renders, as it can take up to 15 minutes until a pixel will appear on the screen, as they trace millions of rays at the same time."
None of the things you claim in this sentence are true. Disney does not use PRman at all anymore.
Sorry, I guess what I meant was that Hyperion is perfectly capable of being used in test renders. The amount of time before you see any resolved pixels is of course dependent on the size of your scene, the desired resolution, etc. It could take 15 minutes before a pixel shows up, but that's not a standard case for production test renders in an artist's daily work.
Because until last year when Pixar released PRMan 19 RIS, PRMan was not that good at ray tracing / path tracing - the REYES method of rendering and shading didn't scale as well for big complex scenes when global illumination and raytraced light occlusion was done.
The RSL shading language was also causing a rather large overhead of shading time as it was an interpreted language that could no longer amortise its runtime cost over a shading grid of points using SSE - in path tracing, you generally shade one point at a time per ray vertex, although you can batch them up in some cases (but without significant ray re-ordering and deferring, after the first bounce this becomes less and less useful).
Disney also use Ptex for texturing, which doesn't really like incoherent shading points, so that was another reason to put effort into doing large scale re-ordering of rays and batching them up.
The REYES algorithm isn't really the problem, you can just generate an acceleration structure and shade each shading point using ray tracing [1]. The REYES algorithm is just an algorithm that generates shading points and determines in which pixels those points are visible, it puts no restrictions on how you shade those points. The downside to mixing REYES and ray tracing is that you have to keep two representations of the geometry in memory, but that doesn't mean you can't combine the two.
The main problem with RenderMan before version 19 was that it didn't have a good acceleration structure and that you basically had to implement a path tracer in RSL on your own. Pixar wrote a path tracer in RSL for Monster's University, so Disney could have just used that. I doubt that they would write their own renderer if they could have just asked Pixar to implement a better acceleration structure in PRMan, so there were obviously different reasons too.
I think Ptex was an important reason for them to write their own renderer, as they seem to put a lot of focus on textures in their paper on the architecture of Hyperion [2]. Hyperion's architecture also allows them to use packet tracing and to load and subdivide geometry on demand, which is also really great.
It was the problem, because doing REYES hiding you suffer the huge "overdrawing" issue, where you need to set the shading rate per object in the scene based on its size on screen such that it doesn't get too much shading. If it had too much shading (the shading rate was too low), many more rays will be sent. And this is because REYES shades stuff before it sees if it's visible, whereas raytracing from the camera (with the raytrace hider in PRMan) only shades the points which are visible, so it scales much more on complex scenes (when GI comes into the picture - without GI REYES can cope easily as it pages everything).
This article[1] implies that there's a lot of cross-pollination with regard to computer graphics research and technology within the movie/VFX industry. So my guess is a bit of friendly competition that keeps spurring new innovation (similar to the browser wars).
Not backed by an article, but I'd assume there's also the factor of needing a new or different capability for an upcoming project, which would prompt the technicians to try something different without affecting in-flight projects. They can always share some of Hyperion with Pixar when Pixar is at a point to change things.
This is mostly due to the two companies being run completely separately. Resources are not shared between the two because Disney Animation is unionized where Pixar is not. It can get messy when you take non union work from Pixar and use Union work at Disney Animation and vice versa.
The real answer to this is basically that the two companies aren't all that integrated, even today. At the same time, renderman was still using ray tracing as part of it's hybrid REYES architecture.
Even so, they could have made it with renderman. Presentations saying 'this wouldn't have been possible' are more marketing than reality. It was more likely to be a combination of the right time to pitch creating a new ray tracer to see how much could be gained through batching while also having a renderer they could control and make sure was modern.
They talk about the bounces of light, which reminds me of radiosity global illumination[1]. I'm left wondering if this idea of going "from the camera to the light sources" is applied in any particular radiosity implementation, and which are it's differences in respecto to the basic form of the algorithm (that would be "from the light sources to the camera").
Radiosity doesn't track where light is coming from or going to, just the total amount of light hitting a face from any direction. It's built around the simplification that all surfaces are perfectly diffuse so that it can iterate through all the faces and exchange light with every other face.
This has its advantages and disadvantages, the biggest advantage being that it's "view independent." Since there's no specular reflection, a surface looks equally bright from any direction, and you don't need to recalculate anything if you move the camera. The big disadvantage is there's no soecular reflection, so light bounces wrong off anything that should have been glossy.
I'm not aware of any attempts to do it "backward," and I can't think of how that would work. The camera in radiosity is essentially an afterthought; you can calculate the entire scene without even having one, and the view you get afterward is more of a data visualization of that result than anything that involved the camera in a fundamental way.
Sounds about right. Not having the time to work on implementing this sort of thing (even as a toy project) leaves me at the point where I sometimes read about the algorithms but can't see through them into the details.
In the first comparison I see that the shadows in the picture generated by Hyperion are way too fuzzy compared to the reference photo. Is this a problem of the technique or just the result of the way in which lightning and polygons have been set up?
This web page is from this month, not 2013. They published a paper on the architecture of Hyperion in 2013, but that never received a lot of attention as it appeared out of the blue and it was published at a 'minor' conference. As most people thought that Disney was still using RenderMan, people never realized that they were actually using that architecture in a production renderer. A few months ago FXGuide published an article on Hyperion and in the past few months there have been a few talks by Disney on it, but this is the first public article by Disney specifically about their renderer. And besides, there aren't a whole lot of discussions on Hacker News on offline rendering, so this article is a good opportunity to discuss graphics.
From the start the art team on Big Hero 6 wanted to make a really big movie. They envisioned those big sweeping shots of San Fransokyo that made the technical folks shudder. They have a massive rendering farm at their disposal but still probably wouldn't be able to render all the shots in time for the release; for one reason, because the art folks kept making changes, and for another, because they contain so many entities. A handful of folks on the rendering team built a proof-of-concept renderer with support for the global-illumination look they wanted while also being able to handle the massive scale.
As the linked article mentions, the solution was to optimize a massively-parallel algorithm by finding coherent rays that can be efficiently calculated together. It's a batch method: start casting a bunch of rays, identify similar rays and group them, calculate collisions for each group, cast reflected rays, identify similar rays and group them, etc. What I like about it conceptually is that in a way it treats light as a field rather than as individual directed rays.
Their initial tests were rendering an infinite plane of generated buildings. The dystopian metropolis was mesmerizing, and I think Disney would be wise to come up with some dark plot line to set there. Anyway, someone way at the top (the director or art director I think) saw the proof-of-concept renderings and decided that it was the perfect tool to attain their vision of this movie, which, by the way, has a hard release date something like three months away.
So they proceeded to turn the proof-of-concept into a production-grade renderer in time to render all the frames and save the movie. And somehow they pulled it off, they released a really beautiful movie, and they wrote a paper about it, too.
[0]: http://www.acm.uiowa.edu/uicc/speakers.html#nichols