I look after a hand full of Linux servers, and as good practice I always make sure I used very complex passwords for console access and change them regularly.
However I appreciate that even the best passwords in the world could in theory be cracked eventually by pure brute force over an SSH connection. It may take a while but it could happen. The more obvious risk however is that if a possible intruder gets lucky.
I’ve used a great little program called Fail2ban for the last couple of years, what this does it quite simple, you can configure how you like, but in essence if someone tries to logon to a server and gets the password wrong a given number of times then it’ll lock down the firewall from that IP. I see it a bit like tarpitting, it won’t stop them but by heck it’ll slow ’em down.
The functionality ‘out of the box’ for Fail2ban is pretty good, you can apply it to your mail logs to block spammers who fall foul of another filter, you can set it up on your web server to stop HTTP scan attacks, ditto for FTP and even SAMBA.
In the past I’ve used it to mitigate the affects of a DOS attack on a webserver I was asked to have a look at. Great program and a fantastic little tool that should be on every sys admins radar.