My experience with stuff with C-extensions (although not specifically ML packages) is that there is almost always a wheel published for them these days, so pip works just fine and I almost never have to think about if something is pure python or not.
I wish I could say I never had to think about it, though.
I consider it a public service to keep mentioning this though, so that in future jobs, I don't have to explain over and over again why I don't want to use pip to "manage" our dependencies.