Coming from DOS/Windows world, I think the main reason ELF is as weird and complex as it is, is because it tries to be both a dynamic and static object format at the same time, and dynamic linking on Unix-likes is also supposed to be an approximation of static linking, but at runtime. In contrast, PE exports are just a mapping from addresses (more precisely, RVAs) to names, and imports are simply an array of slots that point to the name but get overwritten by the actual address by the loader.
An array of slots that point to a name is actually a description of the ELF PLT (procedure lookup table), which is how it does dynamic relocation. The difference is the PLT usually points to the GOT (global offset table) not the name, because in a running program names are irrelevant and only addresses and offsets have meaning.
Resolving symbols through the PLT/GOT mechanism is much faster than resolving through string lookup at program startup (or library load). That's one of the main reasons why program startup and library loading is measurably faster on an ELF-based system than on a COFF-based system like Microsoft Windows.
It gets a bit more complicated than that as DLLs symbols are namespaced, can be changed to lazy link on demand instead of process/thread start, can represent .NET code instead of native, can contain COM objects with dynamic configuration and lookup rules, have lifetime logic associated to processes/threads, can have a special main entry point and invoked via RunDll or regsvr32.
Ah, custom sections and loaders can also be defined.
Stuck in Windows 3.x days? Most of that isn't needed for quite some time.
Also, COFF isn't Windows only, it originated in the IBM world, and AIX is notorious for being a UNIX that doesn't favour ELF by default, even thought it supports it as well.
