Wandering Thoughts archives

What limits the number of concurrent connections to a server

Suppose that you have a socket-based server/service of some sort and you would like to know something about its load limits. An easy but artificial limit is how many true concurrent connections your server can support before would-be clients (whatever they are) start getting connection errors.

The obvious but wrong answer is 'the number of worker processes (or threads) that you have'. This is what I automatically believed for years (with the result that I generally set very high limits on worker processes for things that I configured). In fact it turns out that there are two different answers for two different situations.

If the initial arrival of all of these concurrent connections is distributed over enough time that your worker processes have a little bit of time to run code, in particular if they have enough time to get around to running accept(), the limit on the number of concurrent connections is the number of workers plus your socket's listen(2) backlog, whatever that is in reality. You don't need a lot of workers in order to 'handle' a lot of concurrent connections, you just need a big enough listen(2) backlog. If you're running into this, don't configure more workers, just increase the listen backlog. The time to configure more workers is if there's more work that can be done in parallel, ie if your CPU or disks or whatever are not already saturated.

(This doesn't apply to situations where the worker processes are basically just ways to wait on slow external events such as DNS lookups.)

If you really have N clients connecting to you concurrently at the exact same moment, the real safe limit is only the listen(2) backlog. This is because if all of the clients connect fast enough, they will overwhelm the kernel-level backlog before your code gets a chance to accept() some of them and open up more backlog room. It follows that if you really care about this, configure your listen(2) backlog as high as possible.

Of course this doesn't mean that you can actually service all of those concurrent connections very fast, or even at all. If your worker processes are slow enough the clients may time out on you before a worker actually accept()s their connection. However this will be a user level protocol timeout; the clients should not normally experience connect() timeouts because their connection will have been accepted by the kernel well before you call accept().

In my view it follows that your software should default to using very large listen() backlogs unless you have a strong reason to do otherwise. My previous habit of picking a small random number out of a hat was, in retrospect, a bad mistake (and leaves me with a bunch of code to slowly audit and fix up).

(Since I've been making this mistake for years this is clearly something I need to write down so I grind it into my head once and for all.)

Sidebar: why concurrent connections is artificial

I call concurrent connections an artificial limit because in the real world you generally don't have a pool of N clients repeatedly connecting to you over and over but instead a constant flood of new clients connecting (and sometimes) reconnecting at what you can simplify down to a fixed N-per-second arrival rate. 100 new connections every second is not the same thing at all as 100 concurrent connections; the former is much more demanding. How many arrivals per second you can handle is fundamentally set by how fast (in parallel and on aggregate) you can service clients. If you can service 200 connections per second, you can stand up to an arrival rate of 100 per second; if you can handle only 50 connections a second, no feasible concurrent connections limit will let you handle a sustained surge of 100 connections per second.

(In this situation your connection backlog builds up at 50 connections a second. In ten seconds you have a 500 connection backlog, assuming that none of them time out. In a minute your backlog is up to 3,000 connections, assuming both that your backlog can go this high and that some of the clients haven't started timing out. Both assumptions are probably false in practice.)

(Ab)using awk on the fly

Suppose that you have a file with lines of the form:

host1:   package1 package2 package3
host2:   package3 package5
ahost3:  package1 package2 package3

You want to transform this into something that looks like:

package1 package2 package3
   host1 ahost3
package3 package5
   host2

In other words, aggregate together all of the hosts with a common set of packages (in this case, packages to update that require manual work).

One of the problems of modern Unix is that there are simply too many programs that do random chunks of text processing for anyone except a specialist to remember or even know them all and know what they do. Thus it's quite possible that there any number of clever ways to do this with relatively standard and widely available GNU or other tools. I just don't know what they are off the top of my head and it is much faster to use tools that I know, even in brute force ways, than to go searching and searching and maybe not find anything.

So here is how I did this, on the fly, using tools that I'm already familiar with (which primarily means awk). Let's assume the file is pkglist:

sort -b -k2 pkglist | sed 's/: */:/' |
  awk -F: '$2 == last {sum = sum " " $1}
           $2 != last && last {printf "%s\n\t%s\n", last, sum}
           $2 != last {sum = $1; last = $2}
           END {printf "%s\n\t%s\n", last, sum}'

(The actual version I used put all of this on one line, because a nice clean multiline thing isn't the kind of thing you do on the fly; it's what you do when you're cleaning it up to write about.)

The 'sort -b' bit is due to a GNU sort gotcha.

(Yes, I really write this sort of complex thing on the fly.)

The meaning of listen(2)'s backlog parameter

I was all set to write an entry where I was going to mention in passing something about the effects of listen(2)'s backlog parameter. Then I made the mistake of actually testing things to see if I understood things correctly. Now I am writing a different entry.

If you read your typical Unix manpage for listen(), it will say something like:

The backlog argument defines the maximum length to which the queue of pending connections for sockfd may grow.

Ha ha. As if. In practice the only portable assumption that you can make is that the backlog sets the minimum number of connections that may be pending, where the minimal definition of 'pending' is 'the other side has called connect() and you have not yet called accept()'. The actual observed backlog that can build up depends on the specific Unix version you are using and on at least some Unix versions, on the specific type of sockets involved (on Linux, Unix domain sockets behave quite differently from TCP IPv4 sockets). What happens when there are too many pending connections is also system and perhaps socket type dependent.

'Pending' may also have various different definitions. The minimal one is 'the other side has had connect() succeed', ie the kernel has successfully accepted and established the connection on your behalf (cf); it's just waiting for you to get around to call accept(). For TCP connections, incomplete TCP connections (ones that have not completed the three-way handshake) are not counted against this limit on basically any modern Unix. However, your kernel may not complete the three-way handshake if doing so would make the pending queue too high.

(In the distant early days of the modern Internet, Unixes did count incomplete TCP connections against socket backlog limits (for good reasons at the time). Then people discovered that SYN flooding was a great way to DoS servers that they didn't like. Nowadays any serious Unix has much better protections against SYN floods that don't chew up resources and so don't need limits.)

Note that most Unixes silently impose a maximum value on the backlog value. Some impose a minimum as well, and some impose different minimums depending on what the socket protocol is.

This leads to the obvious question: what should you set the backlog value to when you call listen()? My current view is that setting it low doesn't reliably do anything so you probably should set it high. Given that the kernel will clamp the value if you go too high I suggest going for at least 128 or 256, maybe more; going high in your software means that the actual number is up to whatever limit the local system has set.

(There is an important consequence of setting it high, but that's the entry I was originally planning to write before I made the mistake of checking reality.)

By the way, all of the system and socket type dependent behavior of listen()'s backlog means that you should never trust tuning advice about what you should set it to from anyone who doesn't specify their Unix, the specific version of their Unix, and the specific socket types involved. If someone says 'oh, you want to set it to 64' without giving you that information (and without any other explanation), you can ignore them. Possibly this means you should ignore me too.