You'll most likely be ok. The main attack vector is a chunk of regularly used usernames and a small selection of passwords. These are quite successful against shared hosting where password quality is hard to control properly.

If you've got a "garden variety" server with a strong password, I wouldn't worry.

I've got a laptop slung on the end of my ADSL line that has had literally millions of attempts.

If you are worried, you can install fail2ban which will block repeated attacks at the firewall level.