By looking at the size and bellies of construction workers, farmers, and people doing all sorts of jobs with significant physical activity, one cannot find much support for this hypothesis.
Back in the day, (almost) everybody was not fat, from the academically inclined to the construction worker. Today, many are overweight, from the professor to the agricultural worker. There used to be more walking, which increased caloric expenditure by 300-700 kcal per day, although sport and recreational physical activity was limited to the young.
The main problems have been the easy availability of cheap and tasty calories, combined with a surprisingly low resistance to the ingestion of those calories.