The core of modern Unix shared libraries
The fundamental Unix development that enabled modern shared libraries
is shared copy-on-write
mmap(). That's not necessarily obvious, so
let me walk through the logic of this.
There are two problems with shared libraries, with the second stemming from the first. The first problem is how to get them loaded into the process's address space at all, and it has many possible solutions. The easiest solution is actually to have the kernel do it, since after all the kernel is already mapping code and data from the executable; all it needs to do is map some additional things from another file (or several other files).
(Kernel loading was actually how some early shared library implementations worked, but that's another entry.)
The second problem comes from the first problem, and it is how to deal with shared libraries being mapped into different places in memory in different processes while still sharing as much physical memory between processes as possible. Here there are many fewer solutions and most of them are not very good (position independent code has limits, for example). The only really good solution is to fix things up by applying runtime relocations to the shared library, using copy on write to de-share only pages that needed relocations.
In theory the kernel could do this relocation as it loaded each shared
library into your process. In practice no one wants to have that much
complicated code running in the kernel, so the relocation needs to
happen in user space. The simplest way to do that is to have user space
handle the entire job of loading shared libraries, and to do that user
space needs a way to set up those shared copy-on-write mappings for
shared libraries, ie it needs
mmap() or something very close to it.
(Doing all shared library loading in user space also allows for all sorts of useful flexibility and powerful features that would be awkward or out of place in a kernel implementation. For that matter, it allows the implementation itself to be replaced.)