At night it is harder to see food. It is harder to see predators, some of whom are in fact nocturnal. It is harder to notice visual cues and gestures from allies/kin. It is harder to navigate, both due to difficulty seeing distant landmarks and nearby obstructions, so you are more likely to get lost and/or injured. It is colder so your body has to spend more calories to keep you warm.

There are adaptations that can improve nocturnal capabilities, but these typically come with tradeoffs that make diurnal life harder. Evolution is a series of many baby steps - either you need to adapt to not sleeping while you're still at a disadvantage at night, or you need to adapt to being awake at night while you still need to sleep. Neither path seems like it would have been advantageous to our ancestors.