Or not having a relationship or even much of a social life. I've gone through periods when I was single, spent much of my "me time" working on side projects, some of which shipped and made a little money and another one that got funded. Both OP and one of your replies say this is a "choice", which is true. You can have kids and choose to try to spend less time with them if you are not their primary caregiver. You can not spend time commuting or work on side projects during commute if you aren't driving if you choose and get the right job, etc. I think at some point for most people spending time with your kids or partner is more value than fucking around with side projects and "trying to start a business" that doesn't go anywhere. Sometimes you have to build something, and I think in that case you can figure out a way to do it, but just as often its aimlessly building vague ideas that don't ship or go anywhere. Burn out and mental fatigue are also real risks if you push this too far. Although side projects are fun, essentially working two or one and a half jobs for a long period of time (work, eat, work, sleep, repeat) with no real down time or social life will probably burn you out.

edit: That said, I still suggest you don't quit your day job. With the caveat of you should quit your day job if you have savings\runway\revenue\funding to do so, and you'll be ok with your project going nowhere and returning back to a regular day job after that runway expires.