Different manufactures gave different reasons, but in general you should always go by sight and not a ∞ mark. Even in a manual Nikkor lens there will be a shift based on temperature so you don't want to strictly go by the numbers (which is why ∞ is often juuust past focus, it lets you work in more environments).
> Do not trust "infinity" on your lens, it's not accurate!
It's perfectly accurate on the manual focus Nikkor AI-S lenses I use. Can't speak for other lenses. Of course, they don't do lenses like that anymore.