One of the stories that you could tell about the X Window System and by extension graphical Unix is that first came (simple) window managers and xterm. Only later did Unix developed desktop environments like GNOME and KDE. This is certainly more or less the development that happened on open source PC Unixes and to a certain degree it's the experience many people had earlier on workstation Unix machines running X, but it's actually not historically accurate. In reality, Unix has had full scale desktop environments of various degrees of complexity more or less from the beginning of serious graphical Unix.

(This origin story of desktop environments on Unix is sufficiently attractive that I was about to use it in another entry until I paused to think a bit more.)

In the beginning, there was no X. Workstation vendors had to create their own GUI environments, and so of course they didn't just create a terminal emulator and window management; instead, they tended to create a whole integrated suite of GUI applications and an environment to run them in. One early example of this is Sun's Suntools/SunView, but SGI had one too and I believe most other Unix workstation vendors did as well (plus I believe CMU's Andrew had its own desktop-like environment). When X started to win out, these Unix vendors didn't abandon their existing GUI desktops to fall back to a much less well developed window manager and terminals experience; instead they re-implemented versions of their old desktop environments on top of X (such as Sun's OpenWindows) or created new ones such as the Common Desktop Environment (CDE).

Open source PC Unixes didn't follow this pattern because in the 1990s, there were few or no open source desktop environments. Since X window managers, xterm, and various other graphical programs were free software, that's what people had available to build their environments from, and that's what people did for a while. In this, the open source PC Unixes were recapitulating an earlier history of people running X on Unix vendor desktop workstations before the vendor itself supported X on their hardware and had built a desktop for it (and X itself started out this way, as initially distributed out of MIT and Project Athena).

(Some people then continued to use a basic X environment because they liked their version of it better than the Unix workstation vendor's desktop. Sometimes such a basic X environment ran faster, too, because the vendor had written a bunch of bloatware.)