Not that people will do it all the time (I certainly wouldn't) but the charging case thing is there for a reason. Most likely the idea is you keep it somewhere handy (desk, backpack, laptop bag, etc.) and so you put the AirPods in it while not using them.

As far as them falling out, like on the subway or something, that'll be the problem.