Housing is produced with the cheapest labor possible, i.e. immigrants. If you remove the cheap labor we have exploited for decades, then the price of housing will go up, which means less housing will be built.

This isn't just a stereotype. Look at construction, how many appear Latino to you? Domestic people have much higher expectations and are more resilient to exploitation - they won't break their back for super low wages.