The uncertain question of how much RAM our servers need
We're in the tail end of upgrading a lot of our servers to Ubuntu 22.04, where 'upgrade' means 'build a new version on different hardware and swap it into place' (which has sped up our server turnover a bit). Since we're building up new hardware, one of the questions has been if we should put more memory in any of these servers or if the current baseline of 8 GBytes is enough.
Even with a metrics system, this isn't a straightforward question to answer, and in fact we haven't tried to do any sort of systematic assessment. A lot of the time we've shoved more memory in servers more as a precaution than anything else; at our scale, it's not particularly expensive to overshoot on a few servers, while if we undershoot we'll have to have a server downtime to add more RAM. The initial deployment is the one time when the choice of RAM amount is basically free.
(At least on Linux, there are various memory statistics that will give you some idea at the user level, but they can be hard to interpret. More generally, they don't tell you how much memory you may want under unusual or extraordinary conditions. For some servers you may care much more about having them stay up under high load rather than minimizing RAM under normal load.)
On the one hand, many of these servers spent a lot of time with less than 8 GBytes of RAM, in their previous incarnations; our past generations of servers had baselines of much lower memory usage. On the other hand, new versions of software seem to keep growing their memory usage and we're doing that when we move to Ubuntu 22.04. Plus, lack of memory in servers may have been hampering performance in ways that weren't entirely obvious, for example by forcing too little disk cache. On the third hand, there's a part of me that still feels that 8 GBytes of RAM is a huge amount that normal servers shouldn't need.
(One reason I'm thinking of this now is that I'm about to deploy a 22.04 version of our main web server, which now will only serve plain pages (hopefully with the more efficient Apache event MPM instead of the prefork one and run a few user CGIs every now and then. Does this need more than the baseline 8 GBytes of RAM? Probably not, but then we could have a crush of CGI usage one day, if someone writes a memory heavy CGI and it gets linked to from somewhere popular.)
PS: My impression is that people who use containers probably have a much better handle on their workload's memory usage than we do, because I believe that modern container systems want you to configure that reasonably accurately for each container. Presumably people have developed ways of testing their containers to determine their memory usage under load and so on. For us this is too much work, especially since we only have a few reasonable options for RAM sizes (in our current generation, basically 8 GB, 16 GB, and 32 GB, with 64 GB a possibility if we really need it).