Carter's party controlled the legislature for a half century roughly centering on his presidency; it's hard to see why his reelection would have drastically changed policy. Democratic hegemony at the time was strong enough that Republicans were said to be "in the wilderness", which I think might be the origin of that idiom with respect to national politics.
Idk about the US, but “wilderness” as a political idiom is at least as old as Churchill, who is said to have been in his “wilderness years” from 1929-39.
