And compiling in a debug mode also results in huge recompile times. Is it such a problem?

There is no big difference between building with contracts and logs on (log level is selected dynamically, no need to recompile) and building with debug symbols/suppressed optimisations.