> I bet the compile times would improve if it wasn't header only.

If any hypothetical compilation time problem concerns you then rest assure that C++ enables you to develop submodules that wrap and instantiate your templates, eliminating the need to recompile them every single build.