I'm super curious - have you considered getting out of RDBMs totally for this use case? I have personally wondered about graph databases or something different for the backend storage. The problem of storing these deep and dynamic hierarchical relationships keeps leading to what I'll call "weird-fit" solutions. I can't decide if that is a design smell that a different solution would be better or if this is just one of those messy problems that you have to deal with sometimes.

BTW, if there a rule somewhere that all of us who get into grid and batch processing must write our own workflow management systems? We all seem to do it . . .