Recently, while conducting a review at a community college where I'm helping out with security, we tried to determine why they kept having remote logins to generic student accounts, even though they had just reset all the passwords and erased all the home directories, to prepare for a new course.

The probably good news is that they noticed. The not so good news is that they couldn't figure out what was going on. Turns out that some of the student home directores contained a .ssh folder, which in turn contained an authorized_keys file. Some of these files were from as far back as 2011, and others contained a dozen or more public keys. Nobody had any idea of where the corresponding private keys were, and who had access to them, but it is safe to assume that former students had added a key of their own, and some of them had kept occasionally using the lab accounts.

When the teacher prepped the systems for a new set of lab exercises, he reset all the passwords, and erased all the contents of the home directories.  All the contents?  No. The cleanup script that they used did a for-loop over the existing home folders, and basically ran

cd /home/user1
rm -rf *
cd /home/user2
etc

which .. to their surprise .. left the ".ssh" folders and its contents in place. [Why? Well, "*" is a shell expansion, and customarily ignores "hidden" files that start with a "dot". But that's a different story :)]  Due to this, all the authorized_keys remained, and the corresponding keys continued to work, no matter how often the administration reset the passwords on the student lab accounts.

Oops.

Once we had this figured out and resolved, we wondered how many more stale authorized_keys files they had elsewhere in their Unix environment, so we went hunting from box to box:

find / -name "authorized_keys" -exec ls -ald \{\} \;

The advantage of doing it this way is that the "ls" command conveniently lists both the file size and the file timestamp, like thusly:

-rw------- 1 theowrig users 605 Apr 25  2008 /home/theowrig/.ssh/authorized_keys
-rw------- 1 stuhouwe users 393 May 15  2010 /home/stuhouwe/.ssh/authorized_keys
[...]

which gives a first indication of possible issues. While user "theowrig" was still working at the college, he had no idea what or where the corresponding private key was. No abuse of this particular account was detected when we reviewed the logs, but the account basically was a sitting duck since 2008 if anyone else once had access to the corresponding private key.

Long story short: If you are running a Unix environment that permits key-based SSH login, maybe it is a good idea to check for stale authorized_keys files. You can go by date and file size for a first triage, but often will find that you need to look at the file contents themselves for an indication whose key in the past might have been authorized to do what.

Another worthwhile exercise is to check the SSH logs for as far back as you have them, and to extract which accounts are being connected to by means of key-based login:

debian:/var/log# cat sshd | grep -i "accepted publickey" | perl -pe 's/.*for (\S+) from (\S+).*/$1 $2/' | sort | uniq -c | sort -rn
   1008 rendwras x.x.63.79
    550 sablythe x.x.25.12
    263 markraji x.x.25.235
    223 rendwras x.x.10.141
    211 arfranci x.x.65.90
    [...]

This example shows user "rendwras" as a frequent user of key-based login, and he is accessing from two different IPs. Unfortunately, SSH by default does not log the fingerprint of the key that is being used, but even without this information, a quick tally of the logs like shown here can help to spot issues. If you have any other tips on how to keep tabs of private and public keys authorized for ssh login, please share in the comments below.