My assumption is that all economic growth has some component that is based on matter and energy. Even if it's just firing of a neuron inside consumers head.

If that assumption holds, growth is limited by matter and energy. Assuming that you consume all matter and energy and leave behind maximum entropy (thermal radiation) the growth is limited by the area of an expanding sphere.