I'm guessing it's related to segregation, specifically public pools. Swimming is commonly learned in a pool, so if you don't have access to one it would be hard to lean.

Obviously it's pretty outdated, but stereotypes are rarely concerned with accuracy.