In theory, a nice proper network is supposed to have only a single IP(v4) subnet running over any given network segment. But this is not actually fully required; if you're sufficiently perverse you can run multiple subnets over the same physical network. Some people are now asking how on earth you would ever get into such a crazy situation. Well, sit back, I have a story for you.

Suppose that you use private subnets and in particular you put each group in your organization in its own subnet (around here, a group is a research group). Often when you start doing this, you will give a group a /24 (because that's the simple approach). But the thing about groups is that sometimes they, well, grow. Some energetic professor will get a grant for some equipment here and another one will get a grant for a cluster there and before you know it, a /24 just isn't enough space.

In a perfectly ideal world you would have allocated all those initial /24s such that they could grow substantially, at least into /22s. In a less than ideal world you just allocated the /24s sequentially and there is no room to expand them. No problem; private IP address space is capacious, so you can just start over by giving a group, say, a /16. But this new /16 is not contiguous with the group's old /24; you can't just expand the segment's subnet mask and be done.

In an ideal world you could arrange with the group to have a flag day for the changeover between their old subnet and their new subnet; on that day, all machines would go down, all IP addresses would be changed, and the group would completely migrate from their /24 to their /16. In the real world the group is going to laugh politely at you when you propose this, because they (rightfully) have much more important things to do with their time than go through a large disruption just for your convenience. Instead you're forced into the obvious step: you just add the new /16 to their existing network segment alongside the old /24. The group will put new machines in the /16 and migrate old machines from the old /24 to the new /16 at their convenience (generally very slowly).

This does have some small drawbacks. The largest one is that all of the group's traffic between their two subnets is making an unnecessary round trip through your router (and if your router is a firewall, through your firewall rules; you'll wind up making a special exemption for their internal traffic). Hopefully there won't be much of it; if there is, you can sometimes use it to motivate the group into moving some machines.

(The obvious workaround for heavy-traffic machines is to give them a second IP address alias on the other subnet, so they know that they can reach it directly.)

Sidebar: easy IP address assignment in the new /16

We generally suggest to groups that the low /24 of the new /16 be reserved for a one to one mapping of machines in the old /24. Groups don't necessarily have to convert (or dual-home) machines this way, often they renumber them during a move in order to (re)organize their network, but it makes it easy to do a quick conversion or a dual-homing; just change the network without changing the host IP.