It is very much a product of racism. Check out the book “The Color of Law” that covers this issue extensively, and even beginning with the Bay Area.

Anecdotally, when my parents bought their house in the east bay, it came with old (now invalid) covenants that literally said no black people could live there unless they were servants. The house was built in the 40s!