We need a new core router (among other things) and we'd also like to move towards 10GB Ethernet in our machine room. In general there are two broad approaches in this situation: you can buy a (small) router and then various separate switches or you can buy a single big core router+switch unit with many ports. There are various issues involving either choice and in the past we've gone back and forth between them in our network design. During discussions about this today I had an obvious in retrospect realization about our specific situation.

Our equipment funding is quite erratic and as result we buy stuff mostly in bursts. In this environment the problem with a single big unit is that significant updates to it are likely to be quite expensive and you probably simply won't have that money all in one lump at once. Or to put it another way, you may very well not be able to do a slow rolling upgrade with a single big unit. With separate small pieces of hardware you can do piece by piece replacements as you get much smaller chunks of money over time; first you update a switch here, then a switch there, then you replace the (modest) router, and so on.

(Big units like this are often modular but that modularity has limits. After a certain amount of time it's very likely that the vendor is going to stop developing new modules for your chassis; if you want the new whatever you have to upgrade chassis and probably a number of other things as well.)

This should really not have surprised me because it's exactly one of the drawbacks of having a single big server to do a lot of things instead of spreading the same things out over a bunch of smaller servers. Sooner or later you're going to have to replace the big server and that's going to be a lot of money at once. Upgrading the smaller servers may cost just as much (or more), but you can spread that cost out much more.

(In a sense this is a sad thing if the big server or the big core network box offer economies of scale or other benefits. But note that the overall organization may be getting important benefits from being able to spend the same amount of money in a steady but moderate stream instead of very bursty large chunks.)

Sidebar: the spares issue

Another issue with a crucial core router is spares and redundancy. With modular units you can stock a sufficient amount of spare modules instead of fully duplicating the unit, but chassises can break too and a spare one is probably not cheap. With a big unit (even a modular one) you're effectively paying for more spares and redundancy than you actually need.