One of the major projects I maintain at $work is an apache module, which has given me so much grief because of the fact that it gets dynamically loaded (and then unloaded, and then reloaded)
It's been very enlightening, but there's no shortage of frustration.
One major points is that a dynamically loaded library can have undefined symbols that are found (or not!) at runtime. This can introduce a lot of trouble when trying to make a test binary. Especially in apache's case, since any of the `ap_` symbols are only compiled into the executable, and aren't available as a static or shared library.
In that situation, the only recourse that I'm aware of is to supply your own implementations of those functions that get compiled into the test binary (not the dynamic library under test, since that will create a collision when it's loaded)
Another thing to consider is that linux/elf shared libraries can be either dynamically loaded or dynamically linked. This is not true on all platforms, so it's not advisable to lean into it if portability matters.
I recommend reading the ld.so manpage[0] -- in particular it outlines many environment variables to control the runtime linker. Notably, LD_DEBUG can be a life saver.
Lastly, I'll also warn against using any global/static memory in a dynamically loaded library. It can be unloaded/reloaded, and that can create a lot of havoc, especially if a pointer to that memory gets saved somewhere that survives the unload, and then is accessible after the library gets reloaded.
libprotobuf is a major victim of these types of issues - which is why (or a major contributor) to why the libprotobuf-lite library exists.
I have to care about this sort of thing at great length in my current work. Targeting hundreds of linux / embedded environments. I picked up a tip or two and in general this is a very well written overview of the moving pieces I wish I'd found when I first had to dig into this domain
The first time I learned about dynamic loading was while watching an episode of Casey Muratori's Handmade Hero - a series where he builds a video game, in C, from scratch.
He compiles the game code as a dll and dynamically loads it at runtime in the Win32 platform layer code. This way he can keep platform code and game code separate and reload the game code at runtime if any changes are made. Being new to this technique, I was impressed to say the least.
I always assumed the same functionality was available in the linux environment, but I hadn't bothered to look it up. Now I know.
Edit: LD_PRELOAD would have been useful to mention in the article. It's a useful mechanism, e.g. for profiling by hooking relevant functions, like the PMPI mechanism in the MPI standard.
LD_PRELOAD is also a great way to modify programs you don't have the source for. I used to work for a place that use Perforce as their source control platform. Perforce has/had a CLI tool you could use to open specific GUI tools called p4vc[0].
The first time you ran the tool, it'd spawn a webserver in the background listening on localhost:7999 (this is now actually documented, back then I found out via strace). The problem was that we had many people working on the same machine (in different X11 servers).
So whoever was first to run the p4vc on a machine would spawn the background server. Anyone that tried to use p4vc in the future would communicate with a server spawned by someone else. Since the authentication didn't match, it would create a login prompt (on the DISPLAY of the person that owns the webserver, not the person trying to run the command). It was super annoying.
I created a tiny .so file that would intercept 'bind' and 'connect', check the port to see if it was 7999, and if it was, use the uid of the user instead. Otherwise just transparently pass along all the calls. Problem fixed!
I felt like a wizard, one of the few times in my career that "arcane" knowledge came in handy.
Yes, there's a bunch of common implementations which use it, like (Debian packages) eatmydata, fakeroot, datefudge, substitute mallocs. (One implementation of) fakeroot spawns a demon.
Oh, and if you have something statically linked, so LD_PRELOAD is useless, the usual subversion tool is dyninst.
Very useful article! I wish I had read this a long time ago, would've made a lot of things much easier to understand.
I'd like to comment on this:
> When the relocations are complete, the dynamic linker allows any loaded shared object to execute optional initialization code. This functionality allows the library to initialize internal data and prepare for use.
> This code is defined in the .init section of the ELF image. When the library is unloaded, it may also call a termination function (defined as the .fini section in the image).
There are huge problems with this stuff. Are these sections still used? I know you can add functions to these sections with a GCC attribute but I don't know why anyone would do that. Do other languages like C++ use these sections?
Yes they're still used. C++ heavily uses them as you've guessed. Off the top of my head QEMU also uses them as a way to register all the machine variants it supports internally without having big per arch tables in source that everyone needs to constantly have merge conflicts with.
Very helpful article, clear and informative. Understanding at least the basics of how the linker works at a basic level is very illuminating, and for embedded programming even more so. Found this recently which is a nice complement to this article:
Concerning ldd, see the security warning in ldd(1). I'm not sure if it's still an issue in current GNU binutils, but lddtree is supposed to be safe and the tree may be more useful.
One of the major projects I maintain at $work is an apache module, which has given me so much grief because of the fact that it gets dynamically loaded (and then unloaded, and then reloaded)
It's been very enlightening, but there's no shortage of frustration.
One major points is that a dynamically loaded library can have undefined symbols that are found (or not!) at runtime. This can introduce a lot of trouble when trying to make a test binary. Especially in apache's case, since any of the `ap_` symbols are only compiled into the executable, and aren't available as a static or shared library.
In that situation, the only recourse that I'm aware of is to supply your own implementations of those functions that get compiled into the test binary (not the dynamic library under test, since that will create a collision when it's loaded)
Another thing to consider is that linux/elf shared libraries can be either dynamically loaded or dynamically linked. This is not true on all platforms, so it's not advisable to lean into it if portability matters.
I recommend reading the ld.so manpage[0] -- in particular it outlines many environment variables to control the runtime linker. Notably, LD_DEBUG can be a life saver.
Lastly, I'll also warn against using any global/static memory in a dynamically loaded library. It can be unloaded/reloaded, and that can create a lot of havoc, especially if a pointer to that memory gets saved somewhere that survives the unload, and then is accessible after the library gets reloaded.
libprotobuf is a major victim of these types of issues - which is why (or a major contributor) to why the libprotobuf-lite library exists.
[0] https://man7.org/linux/man-pages/man8/ld.so.8.html