This change, while dramatic, isn't necessarily a sign of problems. Over the past 35 years, and especially in the past 10--15 years, the US (and most of the rest of the world) has progressively urbanized; in many areas, it has become the norm for people to live in condos rather than detached houses.
I can't find any relevant data on the US Census website, but I strongly suspect that if you add duplexes, condominiums, etc. into the mix, you'll find that while houses have become relatively unaffordable, housing has not.
I don't believe this is alone because of location.
Building codes in cities are usually considerably stricter than those in the suburbs and ex-urbs, and meeting code itself requires more expensive design choices. Additionally, you're not going to build an inexpensively built, unnecessarily gabled raised ranch with a portico in the middle of a city, mostly because you wouldn't get approval because the aesthetics would conflict with the existing neighborhood.
Price and location are not independent. I looked a while back in an MA suburb where 2300 square foot homes were being sold around $300K. I looked into buying an equivalent-sized empty plot just outside that subdivision on which to build a 1150 square foot "atomic ranch", and I was looking more at $500-550K for something half the size on a similarly priced plot.
Cost per square foot relates to both the building height and land costs. A ten story building on cheap land costs a lot more per square foot than a once story house. However, as the cost of land goes up that shifts.
PS: I spend 1700$ a month on a 1br apartment because it's in a 20+ story building and on expensive land (great location). The problem with apartment buildings is they are only reasonable in expensive areas so their minimum price is even higher than the premium of building the building.
I can't find any relevant data on the US Census website, but I strongly suspect that if you add duplexes, condominiums, etc. into the mix, you'll find that while houses have become relatively unaffordable, housing has not.