Your reasoning is flawed, technically if we make things last longer, be repairable, skip vacations on the other side of the planet and stop buying so much cheap plastic crap we would most definitely "degrow" but the quality of life would at least remain unchanged.
Capitalism is a beast that needs constant feeding, it doesn't do much good in many cases, it's just growth for the sake of it.
The vast bulk of global energy production does not go towards "cheap plastic crap" or vacationing. It goes towards transportation (which affects the base cost of practically all economic activity), industrial uses that are essential to healthcare, construction, and agriculture, and the heating and cooling of our living spaces and places of work. These expenditures will only increase as the world's human population increases to a projected ceiling of around 10 billion. In addition, replacements for "cheap plastic crap" may consume more energy to produce, even if their production emits less carbon.
There are certainly efficiency gains to be had. They will not be made overnight. America's housing stock alone can't be replaced in a week, or a year, or a decade with hyper-insulating, low-emission construction materials. Changing from one mode of construction, consumption, or production to another has an energy cost, too.
Kate Soper makes the case that quality of life would improve significantly if we took these steps. The carbon-intensive "good life" leaves us time poor and stressed.
You can argue how much a change in quality of life it is, but not being able to travel or buy cheap plastic things is certainly a reduction in quality of life.
About transport - we can always switch to greener alternatives. Forbid oil-based flying, and the need to travel will bring new solutions.
Plastic is a crap and poison from the time of manufacture, not enhancement of quality of life. There is nothing (except medical equipment maybe) positive about it.
Capitalism is a beast that needs constant feeding, it doesn't do much good in many cases, it's just growth for the sake of it.