It's good to hear some insight into this, as it's really been grating on me lately, so thanks!
I'm somewhat curious about what kind of problem you could have that would allow the 'live' site to work, but break through CloudFlare. That seems entirely counter to the functionality... do you happen to remember or are you able to comment on what was happening?
Sure. There are at least a couple ways I've seen that it can be triggered (and if one of the Cloudflare guys on HN is around they might have more or correct me if I'm wrong).
The first seems to be that if a page is experiencing slow loading times on the upstream locations (and pretty much everyone using Cloudflare is likely just an 'upstream' entry in nginx), you can trigger the error. If the upstream server returns a 500 error I have seen certain circumstances where that will return a live page error... or, if the web-server on the remote end terminates the connection for whatever reason that will also cause it to happen.
For us the last time we had a problem it turned out to be pretty complicated. There was a bug (which we reported and was fixed in the last release) for the php NewRelic module. Whenever there was what the module considered a 'slow' mysql query, it would cause a segmentation fault. This particular day there was randomly a few 'slow' queries, thus when the segmentation fault occurred the php thread would die killing off the connection and bringing up the 'page is down' error from Cloudflare. That was a pain to diagnose. Strace for the win.
I'm somewhat curious about what kind of problem you could have that would allow the 'live' site to work, but break through CloudFlare. That seems entirely counter to the functionality... do you happen to remember or are you able to comment on what was happening?