Non-potable is the industry term for "non-human drinkable".
And yes, the cost of that would be huge, not just in cities. In the UK the decision the victorians made to not have separate sewage and rain water drainage systems still has a knock-on today and costs a vast amount in water treatment, but the cost of separating rain runoff and sewage is still seen as prohibitive.
edit:
However, on a small local-scale grey water systems can do a lot of good, both for re-use but also rain water capture and re-use would help with flood prevention. If houses captured rainwater for use for lawn watering and other appropriate uses, this is something that could be done without great expense.
Be careful of the law of unintended consequences. Rainwater capture will be fine so long as only a minority do it (how small I don't know) or only a fraction is captured but if everyone does it then it will change the economics of water supply and give people incentives to use the captured water for other than lawn watering. For instance they might use it for flushing toilets or even bathing. the problem that then happens is that the local water table will fall because it is not getting the water that it used to because that water is now flushed into the sewage system.
I'm not arguing against the idea, especially as it is typically implemented in the UK where a house will generally only have about 500 litres of storage, but scaling it up and making it a requirement could cause some interesting problems.
It's unlikely that a given residential area is drawing water from the area directly beneath it, or that its water supply is fed by residential run-off. That's not going to be clean water that you'd want to drink. Most residential water comes from resovoirs fed by mostly unpopulated catchment areas which are often 10s or 100s of miles away. Also bare in mind that around 50% of rainwater evaporates or ends up in the ocean, so there's plenty to go around.
I thought that the idea of rainwater capture was to prevent the water from going to "waste" as runoff. That is, the water that hits the roof of your house ends up in a torrent in the downspouts, which runs into the streets and the sewer system. Whereas the water that hits your yard has more of a chance to sink into the water table.
And yes, the cost of that would be huge, not just in cities. In the UK the decision the victorians made to not have separate sewage and rain water drainage systems still has a knock-on today and costs a vast amount in water treatment, but the cost of separating rain runoff and sewage is still seen as prohibitive.
edit:
However, on a small local-scale grey water systems can do a lot of good, both for re-use but also rain water capture and re-use would help with flood prevention. If houses captured rainwater for use for lawn watering and other appropriate uses, this is something that could be done without great expense.