It's probably your router. The ones supplied by ISPs, as well as many cheap off-the-shelf ones, are terrible at managing load.