Boston is incredibly segregated and in the area I live in is less than 10% white. It's not a coincidence that this is the only area in the city I've ever had trouble getting a ride to (roughly 200 total rides).
Maybe they were afraid for their own safety? I know when I visit the US, each city has it's no-go areas that you want to avoid. I remember my first time in Dallas many years ago pre-GPS era, drove somewhere for dinner, stopped to fill up petrol and get directions to my hotel and the service station attendant told me I should probably exit the area ASAP because it is not safe.
I lived in Boston a couple years before Uber was a thing and if I was out at night and needed a ride back to Southie (where I lived), it would take me four or five taxis at least to get one to agree to go there.
Uber eliminated that problem. It would be terrible to go back to that system where drivers can just refuse you based on where you are going or what you look like.
