This is probably a hard problem, but the dependency graph resolution in conda is a thing of nightmares. It is so appallingly slow even for relatively menial tasks that updating environments becomes an exercise in frustration.
I'm unsure what are the core issues, but in my experience cargo was always pretty quick to use and if it fails, it fails fast.
Conda on the other hand is slow for the simple cases, and if the graph becomes complex it will just churn for 15 minutes before throwing it's hands in the air and giving up with some cryptic error.
I suspect it comes back to the fact that packaging and dependency management was thought about upfront for Rust and the whole ecosystem was built well from the get go?
I've been meaning to write a blog post on this topic. That is, on the various speeds of various package managers. I don't know a ton about Conda, but when I was looking into Poetry, one of the core issues is: https://python-poetry.org/docs/faq/#why-is-the-dependency-re...
> This is due to the fact that not all libraries on PyPI have properly declared their metadata and, as such, they are not available via the PyPI JSON API. At this point, Poetry has no choice but downloading the packages and inspect them to get the necessary information. This is an expensive operation, both in bandwidth and time, which is why it seems this is a long process.
Cargo doesn't need to do anything with packages directly to do its job; everything it needs is in the index. This makes it pretty fast.
>> This is due to the fact that not all libraries on PyPI have properly declared their metadata and, as such, they are not available via the PyPI JSON API. At this point, Poetry has no choice but downloading the packages and inspect them to get the necessary information. This is an expensive operation, both in bandwidth and time, which is why it seems this is a long process.
This sounds like something that could be done server side, either by PyPI or another entity and expose it through a new API endpoint, instead of doing it on every Python developer's machine.
How do you ensure a team of 30 developers are all using the same version of python and all of the dependencies of your project with conda? How do you distinguish prod dependencies from dev dependencies? How do you update and manage transitive dependencies?
Conda manages Python itself as one of the dependencies - so that’s not actually a problem.
I used conda to manage 2.6, 2.7 and 3.3 side by side, a d that was fine. I never locked the patch version (e.g. 2.7.3 vs 2.7.5) though that is definitely possible.
It apparently requires a specific workflow - explicitly editing the requirements.txt file, rather than freezing it - which is not harder and which I was doing since day 1 but is apparently uncommon.
(And it worked well across a few tens of machines, some running windows and some running Linux, with a mix of os versions and distributions. So I know it worked well already in 2013, and I’m sure it works better now).
Speed is not good, but I never had it take more than a minute for anything. Some people here are reporting 15 minutes resolution times - that is a real problem.
You make sure everyone uses the same Python version by setting it as a dependency. I mentioned I only set the dependencies on minor version (e.g. 2.7) rather than path version (e.g. 2.7.3), but the latter is supported - for the python version as well as for any other package.
You make sure the exact versions you want are in use by editing and curating the requirements.txt rather than freezing it. It really is that simple, but somehow that's not a common workflow.
prod vs. dev is the only one I didn't address because I don't have experience with that - but I know people who manage "requirements_prod.txt" vs "requirements_dev.txt" and it seems to work for them.
How do you manage the specific version of transitive dependencies? For example, I install library A and it requires library B, but your code doesn’t use B directly?
A better question, what’s your workflow for installing a dependency and 6 months later updating that dependency?
Just put them in your requirements.txt even if you don’t depend on them directly.
That’s basically the whole idea of conda’s solver, I think. If your list works - fine. If n9t, it will find a set of package installs that makes it work.
I guess that’s also why my resolution times are way faster than what some people describe.
I treat requirements.txt closer to a manually curated lock file than a wishlist (which most req.txt files in the wild are).
This is a giant pain in the ass compared to every other modern language. Either you’re manually managing precise versions for all direct and transitive dependencies or you aren’t running exactly the same versions of libraries across team members and environments which would be horrifying.
There is a command (not freeze) that dumps the exact versions deployed, similar to a lock file. I never used it, because it doesn’t work well across operating systems (e.g. do it on Windows and try to use it on Linux). This is less of a Conda problem and more of a Python problem - same package name/version has different dependencies on different architectures.
But manually curating the dependencies has been painless and works fine for me and my team for almost a decade now.
And they were the biggest headaches on my team, bar none, of them all. An absolute nightmare to coordinate--and it seemed like even minor things that should be unrelated, e.g., a locally installed system (not python) package slightly off-version, could eat a developer's time trying to resolve.
i've never used it particular feature, but then i'm using python since 1.5 and i'm just used to it being a bit behind the times. stockholm syndrome, you might say, especially after trying out rust and seeing the work of art cargo is.
In addition to the other replies, one I've encountered is the case where conda doesn't provide a build for a package (thus, one must use pip or something else to manage that dependency) causing weird issues with respect to maintenance and updates.
The two worst are in the other comments: ensuring sync'd dependencies across multiple environments and developers, and the horrendous resolution times leading to a useless error message when failures occur.
conda has "environments" which are isolated from each other and can each have specific versions of python and dependencies installed in them (and each has their own "pip" local installation as well, in case something wasn't specifically packaged for conda).
Have never used cargo - what can cargo do that conda cannot?