Famously, modern Unixes by limit the (default) size of the main process stack and the size of thread stacks; they pick different limits, which causes issues because C has no way of dealing with this. Today, for reasons beyond the scope of this entry, I became curious if V7 had any stack size limit and if it didn't, when such a limit appeared.

The short answer about V7 is that V7 had no stack size limit. The combination of your stack and your program's data (and perhaps your program's code, depending on the model of PDP-11) could take up all of the 64 KB of memory available to your process for this. If your process took a memory access fault and its stack pointer was below the kernel's idea of the bottom of your stack, the kernel automatically grew the stack as far as necessary. You can see this in the V7 trap.c, where the decision to grow is made, and sig.c, which has the routine for the growing (and also ureg.c for one additional function).

On the BSD line of Unixes, 3BSD doesn't seem to have anything but 4BSD introduces a vlimit(2) system call to get and set limits, including the stack size. By 4.1c BSD, this has changed to the now familiar getrlimit(2) and setrlimit(2) system calls.

(Interestingly, Linux glibc provides a vlimit(), according to the manpage.)

On the AT&T side, there doesn't seem to be anything much that I could find. The System V Interface Definition (SVID) seems to be the official source of a ulimit() system call, but its only official thing is limiting file size. As far back as System III, there is a ulimit(2), but it doesn't set the stack size limit; the closest it comes is that it can get (but not set) the brk(2) limit. The System III VAX trap.c and sig.c still seem to be free of stack size limit checks, and have comments that say things are grown unconditionally.

(The current Oracle Solaris ulimit(2) manual page still talks about being able to get the maximum possible break value, although this isn't required in POSIX.)