I think this is conflating growth with technological advancement. One can grow without technical advance (e.g. farming on more land produces more, feeds more, uses more, etc) or advance technically without growth (using your examples, we can reduce infant mortality and have the same amount of children, defend ourselves better, etc without needing to necessarily use more, generate more stuff)
To the extreme, I sometimes wonder what would happen if we invent replicators. It would obviously be great for the average person, but it would undoubtedly wreck the economy :)
