They were referring to compile times to build boost itself, not of their own projects. On most Linux distros, there are binaries already available so that's not a problem; yes with brew on OSX it takes a long time but you don't do it that often.

Of course template-heavy, header-only libraries will increase compile time, but not insanely so. I try to be careful about including boost's ease-of-use headers which pull in everything and keep it limited to what I really need from a given sub-project.

Most of what I use from boost is in C++11, except asio, but the same principals apply.