I think due to evolving as a social animal we necessarily have a sense of justice and a notion that bad behaviour must be punished or corrected. The notion that all bad behaviour is logged by a mysterious being that will present you with whatever comeuppance you deserve after life is a beautiful fiction. It makes one feel much better to believe that over thinking the more powerful person who took your things by force is going to get away with it.

I think it's more a desire to see justice than a desire for something omni-present. People that believe in neither religion nor good government will sometimes make claims like, "yeah one day I'm going to be that guy's boss," which is just saying, "I feel bad now but he'll get his comeuppance later and balance will be restored."