pam-script with xt_recent works just fine.

Everytime when an authentication fails, you add the ip address to the xt_recent list in /proc and in iptables you just check via --hits and --seconds and then reject the connection attempt the next time.