Letting the mind wander like that is precisely why I read books in the first place. Sometimes I'll only read one chapter in 3 hours and the rest is just daydreaming.