If only one suffered, that would be optimal. In my experience what ended up happening is all suffered. In noticing that one part of my life was starting to falter I would soon over compensate. What would happen is a juggling act, that ends with total burnout. In the mean time you disappoint a lot of people.

Side-projects are good, but you have to be careful about it.