Traditionally, one of the things that the standard login program did when you logged in was to look at the state of your mailbox and report 'you have (new) mail' if this was the case. This worked well when everyone logged in to Unix systems on serial terminals, sort of well when people used telnet and rlogin (both of which actually run login), and went downhill when SSH became common, because SSH does not run login at all.

Clearly the right solution (in the PAM world) was a PAM module to look up this information and report it. Hence the pam_mail module, which is present on many Linux distributions and used by default on at least Ubuntu. (Ubuntu uses it in the PAM configurations for login, sshd, and su, where it is only used to set $MAIL.)

This sounds great, but there's two problems with pam_mail. First, it always attempts to stat /var/mail/<user>; there is no way to turn it off on a per-user basis. If the module is configured, everyone gets this service (this is unlike login, which traditionally respects a $HOME/.hushlogin file). Second, it doesn't just do this when you log in via SSH, which is what you might expect; instead, it does this any time you authenticate to sshd and do something. In particular, running 'ssh host command' will cause pam_mail to try to stat /var/mail/<user> (even though you'll never see the message).

Now consider what happens when access to /var/mail stalls; perhaps it is on a dying disk, or perhaps it's NFS mounted from an NFS server that is having problems. Not only can you not log in to the machine via SSH, you can't even use SSH to run commands on the machine (whether as root or any other user). And all of this is because of an innocent sounding PAM module buried in a PAM configuration file.

Our /var/mail is NFS mounted from an NFS server. You can probably see where this one is going; today, that NFS server went down and havoc ensued. Thus, pam_mail is very much not my friend right now.

(It's also no longer present in /etc/pam.d/sshd on our Ubuntu machines. We're going to tell users who really care about this to put something in their personal dotfiles.)