Traveling would be so much more disorienting. Forgetting what the local daytime range is and checking the position of the sun to guess whether you're closer to the front or back half of the day. Having to consult a table to figure out exactly how much you're going to inconvenience the person giving you a ride from the airport. Accidentally running into rush hour because you forgot that 1100 is 5:00 here. LOL.
So like when you fly from London to NYC and have no idea how long the flight actually is because it leaves at 10am and arrives at 9am? Timezones explicitly make travel harder, not easier.