Adding to my sibling comment, time is also mostly used as a coordination system. Being offset by a few minutes would make aligning meetings with your remote coworker an even bigger nightmare than it is now.