Of course there's a social contract. We also have laws against burglary even if you leave your front door unlocked. There's a limit to how much people are allowed to screw you. If someone takes an action that results in you losing your house, that person is an asshole even if they're not technically a criminal.