Presumably the person who supplies the dice knows how they're loaded. With a shift, they might still be able to use that information to their advantage. Instead, you could use the agreed-upon randomness at the beginning of the game to apply numeric labels to the faces of the dice. As long as the labels themselves aren't loaded... and the person who supplied the dice would still have information about the distribution.