Wandering Thoughts archives

Runtime loading of general code usually requires general dynamic linking

Let me set the stage. You have a system that supports dynamic linking and dynamic code loading in general, with a low-level interface for this; for example, modern Unixes have dlopen(), dlsym(), and so on. You also have a language runtime that's entirely statically linked, and you want to have a program in this language environment dynamically load some of its code at runtime, for example to implement some sort of plugin system. One hypothetical approach to doing this is to directly use the low-level system interface for dynamic loading; you build the plugin code into a single object file, use system tools to transmogrify it into a dynamically loadable one, and then dlopen() it in your program, fish entry points out with dlsym(), and call them.

(Depending on what sort of FFI your language has, you might need some sort of low level shim to sit between your plugin's code and your main program's code.)

All of this sounds nice, but is it likely to actually work if you tried to implement it? My answer is 'probably not'. The problem is pretty simple to state: how does the plugin code get access to the runtime environment and other outside code?

Unless the plugin code does pure computation (without even memory allocation), it needs access to the runtime code that does things like memory allocation, operating system calls, DNS lookups, general IO routines, and so on. Plus, it may also want access to other higher level library code or the like. The plugin can't safely statically link all of this code into itself unless all of the runtime environment is carefully designed so that you can run two (or more) independent copies of it inside the same process, one in your main program and one in the plugin (this is often a big problem for things like memory allocators and garbage collection). Using the runtime environment from your main program requires connecting the plugin to it; you need to either somehow build the plugin so that it knows all of the right addresses in your main program (which binds it to a particular build of your main program) or do your own relocation at dlopen() time. You also need to insure that your main program actually includes all of the code that plugins will need, whether or not that code is used by your main program.

What makes runtime code loading work without hassle is general support for dynamic linking. When the plugin code is dynamically linked, it's already set up to do runtime lookups of outside addresses (and possibly runtime relocation). When you dlopen() it in a dynamically linked program the dynamic loader code only has to connect it to the existing symbol tables in your program (plus any new shared libraries that it requires). And of course a chunk of code that's built to be statically linked doesn't have any of this infrastructure built into it since such infrastructure isn't necessary and in fact is usually undesirable (partly because of the added complexity).