It seems to me that we could eliminate a lot of that overhead if we stopped pretending that Social Security is insurance and just gave every retiree the same benefit. You would still need to keep track of who has reached retirement age, who has died, and where to send the checks, but you could get rid of a lot of arcane rules.
I guess the question is: how much would you save by firing 50k people, and how much would it cost you in claims getting awarded to people without real need?
Firing fifty thousand Social Security Administration employees would likely require the SSA to be the most efficient benefits management organization in the world. 70,000 employees sounds like a lot, but, again: it's roughly in line with comparable commercial organizations.