I've had some discussions with file system developers and a developer of a distributed file system. The sheer number of failure cases that need to be handled, because they are hit in production systems, is simply staggering.