The problem with tone isn't so much that it's hard (not more so than, say, a voicing or aspiration contrast is), but that as people get older, it's harder to hear contrasts that don't exist in the languages you're familiar with.