But they can still be far away that the light hasn't arrived at us yet (or maybe never will?). It's entirely reasonable that space may not be continuous. There may be groups of galaxies far enough away from each other that they burn out before receiving the other's light.
How can we confidently assume this isn't the case?
No, there was a time that all space was hot, and we know from microwave background radiation that it is homogenous. So it’s not just “galaxies are far from each other”, it would take “space time is discontinuous somewhere and this discontinuity is just on the edge of our observable universe”. There is also no older light that can ever reach us, as the amount of universe that is observable is actually decreasing (beyond that, CMB will always be the oldest light as it predates any astronomical object formation). Anyways, CMB really constraints the age/homogeneity of the early universe.
How can we confidently assume this isn't the case?