I can't see how it's simpler, because then you need to write N*M bridges/connectors/plugins to let N package managers interact with M build systems. And what if the library you download is meant to be built with a build system which is different from the one you chose for your application? What if you then want to globally turn on debugging or change a compilation flag, or a preprocessor define to enable/disable a feature?

I don't know of any language-specific package manager that doesn't also make the package immediately available to the programming environment without further fiddling. Solving half of the problem... well, it's a half solution :)