IIRC, it's a speed hack in the IEEE floating point standard.

NaN can happen for a lot of different reasons. You might have divided by zero because your code is broken, or you might have divided by epsilon because your algorithm isn't stable on this problem. [1]

This test checks to make sure none of {f, g, x, y, intermediate_values} were ever NaN, wihtout having to test each one every step of the way.

    if (  f(x) == g(y)  )

Which is a big deal if it's the guard on an inner loop in some numerical code.

[1] I'm fuzzy on the details. Corrections solicited.

If it's just a debugging trick, though, then speed hacks seem less useful.

isNaN() will return true for anything that can't be coerced into a number while !(x == x) will only be true for the NaN value itself.

How do you know isNaN() doesn't use this fact? In Java, the body of isNaN(n) is simply 'return n!=n;'

It depends on the implementation. In C, it is quite easy to have n != n optimized away when compilers have their compiler flags set on. That's why you should use isnan to test nan instead of any other techniques when using C or C++.