If chargers become widely available then this is only a consideration for long road trips. Otherwise your car can be charging at practically any time it's not being driven. Meaning it stays near full battery most of the time you start driving. That's not true for gas; you don't have a gas line connected to your tank in every parking lot.
Teslas have a driving range of ~200 miles or something like that right? After driving 200 miles, most people are happy to stop and stretch their legs for 15-20 minutes. Tesla superchargers take about 30 minutes to charge your battery most of the way. Drive another 200 miles and you're ready for lunch or dinner, which could be a ~1 hour break, if you're not getting drive-thru and a chance for a much longer charge.
The only limiting factors I see are cost, the availability of charging options (and density of locations) and charging for apartment dwellers. Charge times are probably "good enough" already and likely to get better.
Until battery/recharging tech makes that leap, we have plug-in hybrids. My Chrysler Pacifica PHEV minivan gets 30 miles on battery before reverting to a regular hybrid getting 30 MPG.
I think widespread electric vehicle adaption will happen due to price but you have a very valid point on recharge time.
Basically your car is unusable if low and being recharged and there is nothing you can do about it. Unless swapping heavy batteries becomes practical.