Eventually you work less so you never earn that money in the first place. This may involve saving up first so you have more economic bargaining power, or it may involve collective action to reduce working hours across the board.
My only point in referencing saving is that it is one path to achieve the bargaining power to actually work less, against employers who inherently want to occupy your entire life. I'm talking about having saved up enough to be comfortable working less - after which you can turn your reduced spending into reduced working. An alternative avenue that doesn't involve saving at all is collective action, which is how we've gotten down to a 40 hours per week standard.
"The economy" is defined by what is legible in the financial system. In this example, that is working for a wage. Time spent on other endeavors is not reflected in such accounting (sure, money spent on those activities is obviously still part of the economy, but said money is only part of what is spent on them). Time spent on other endeavors is therefore not "growing the economy", except in some pathological sense of trying to shoehorn our entire existence into economic terms.
Time spent with family or on hobbies certainly does make society better though, despite not being economic activity or "growth". In fact, this is exactly what opponents of the growth-worship philosophy are aiming for.
