I have an enduring interest in programs that give you some sort of top-like view of current network traffic, stemming in part from having our NFS fileservers, an active IMAP server, a sometimes quite used web server, all sorts of NFS clients, and so on. In the shiny future where we have Ubuntu 20.04 and 22.04 machines, this interest will probably be significantly met with eBPF based programs. But for now we're still mostly on Ubuntu 18.04 and 18.04 doesn't have good support for eBPF tools, so I have to get by with some old Linux standbys that work through more traditional means.

Iftop displays bandwidth usage grouped by pairs of source and destination hosts (at least by default). The display is a bit confusing and it has a bunch of options, so you (and I) probably want to read the iftop manpage. Iftop will tell you (me) what hosts are using up a lot of bandwidth, but won't identify the processes or users involved. This generally means that it's good on an NFS server to see what clients are active or on an NFS client to see quickly what NFS server it's mostly talking to.

Nethogs displays bandwidth per process, or by TCP connection if it can't identify a process that's responsible for it (so on our NFS clients, it often shows a bunch of traffic to TCP port 2049 on our fileservers). Nethogs is potentially overwhelming in how much it shows but can be useful to pick out a single IMAP process or scp or whatever that's using up an unusual amount of bandwidth. However, nethogs doesn't show the remote IP, just the process; sometimes it can be easier to find what's responsible with the remote IP, so you want iftop instead.

Because I keep trying it out in hopes and being disappointed, I will note that iotop doesn't show NFS IO, only local disk IO. For our systems, local disk IO is generally uninteresting (even on the NFS fileservers, a per process view isn't useful). I'll probably have to write a eBPF program that provides a 'top-N' display of user level read and write IO volume in order to get what I really want here.

(I continue to not have written an nfsiotop, or found one. I did just unearth a 2018 experiment in that direction that I tweeted about vaguely at the time but then forgot about since then. I think I'm going to have to poke at it some more.)

PS: What I'm interested in here is real time, fine grained display of this information, especially on a per-process basis. We already have a general metrics system that captures broad network and NFS client information on a more coarse grained basis.