> None of this example has anything to do with performance or reliance on C dependencies, but ok.
<C dependencies>
You'd realize why I wrote that if you used Java/Maven. Java is by and large self-contained. Stuff like Python, Ruby, PHP, Javascript[1] etc, are not, they depend on system libraries.
So when you install something on Solaris, FreeBSD, MacOS, Windows, well, then you have to deal with the whole mess.
1. Is the C dependency installed on the system?
2. Is it the correct major version or minor version?
3. Has it been compiled with the correct flags or whatnot? 4. If it's not on the system, can the programming language specific package manager pull a binary from a repo for my OS-arch combo?
5. If there's no binary, can the programming language specific package manager pull the sources and compile them for my OS-arch combo?
All of those steps can and do fail, take time, and sometimes you have to handle them yourself because of bugs.
Java is fast enough that almost everything can be written in Java, so 99% of the libraries you use only have 1 artifact: the universal jar, and that's available in Maven repos. No faffing around with wheels or whatnot, or worse, with actual system dependencies that are implicitly (or explicitly) required by dependencies written in the higher level programming language.
<Virtual envs>
I won't even bother to address in detail the insanity that you described about virtual envs, it's just Stockholm syndrome. Almost every other programming language does just fine without venvs. Also I don't really buy that issue with the lack of portability, it's just a bunch of bad design decision early on in Python's design. Even for Windows there are better possibilities (symlinks are feasible, you just need admin access).
I say this as someone who's used basically all mainstream programming language over the years.
The sane way to do virtual envs is to have them be just... folders. No absolute paths, no activation, just have a folder and a configuration file. The launcher automatically detects a configuration file with the default name and the configuration file in turn points the launcher and Python to use the stuff in the folder.
Deployment then becomes... drumroll just copying the folder to another machine (usually zipping/unzipping it first).
* * *
[1] Javascript is a bit of a special case but it's still slower than Java, on average.
> Stuff like Python, Ruby, PHP, Javascript[1] etc, are not, they depend on system libraries.
The Python runtime itself may depend on system libraries.
Python packages usually include their own bundled compiled code (either directly in the wheel, or else building an sdist will build complete libraries that end up in site-packages rather than just wrappers). A wheel for NumPy will deliver Python code that uses an included copy of OpenBLAS, even if your system already had a BLAS implementation installed.
Regardless, that has no bearing on how virtual environments work.
> I won't even bother to address in detail the insanity that you described about virtual envs, it's just Stockholm syndrome. Almost every other programming language does just fine without venvs.
You say this like you think there's something difficult or onerous about using virtual environments. There really isn't.
> Also I don't really buy that issue with the lack of portability
The use of absolute paths is the only thing preventing you from relocating venvs, including moving them to another machine on the same architecture. I know because I have done the "surgery" to relocate them.
They really do have the reason for using absolute paths that I cited. Here's someone from the Pip team saying as much a few days ago: https://discuss.python.org/t/_/96177/3 (Using a relative path in the wrapper script would of course also require a little more work to make it relative to the script rather than to the current working directory. It's worse for the activation script; I don't even know if that can be done in pure sh when the script is being sourced.)
Yes, it's different between Linux and Windows, because installers will create actual .exe wrappers (stub executables that read their own file name and then `CreateProcess` a Python process) instead of Python wrapper scripts with a shebang. They do this explicitly because of the UX that Windows users expect.
> Even for Windows there are better possibilities (symlinks are feasible, you just need admin access).
Please go survey all the people you know who write code on Windows and see how many of them have ever heard of a symlink or understand what they are. But also, giving admin rights to these Python tools all the time is annoying, and bad security practice (since they don't actually need to modify any system files).
> The sane way to do virtual envs is to have them be just... folders. No absolute paths, no activation, just have a folder and a configuration file.
A virtual environment is just a folder (hierarchy) and a configuration file. Like I said, activation is not required to use them. And the fact that certain standard tools create and expect absolute paths is not essential to what a virtual environment is. If you go in and replace the absolute paths with relative paths, you still have a virtual environment, and it still works as you'd expect - minus the tradeoffs inherent to relative paths. And like I said, there is third-party tooling that will do this for you.
Oh, I guess you object because there are actual symlinks (or copies by default on Windows) to Python in there. That's the neat thing: because you start Python from that path, you don't need a launcher. (And you also, again, don't need to activate. But you can if you want, for a UX that some prefer.) Unless by "launcher" you meant the wrapper script? Those are written to invoke the venv's Python - again, you don't need to activate. Which is why this can work:
Again, yes, the wrapper scripts would have to be a little more complex in order to make relative paths work reliably - but that's a Pip issue, not a venv issue.
> But also, giving admin rights to these Python tools all the time is annoying, and bad security practice (since they don't actually need to modify any system files).
You only need to do it once, when the symlink is created...
My point is that venvs are a code smell. Again, there's a reason basically no other programming language ecosystem needs them. They're there because in Python's history, that was the best idea they had 20 years ago (or whenever they were created), where they didn't even bother to do their homework and see what other ecosystems did to solve that specific problem.
> You say this like you think there's something difficult or onerous about using virtual environments. There really isn't.
They're a bad, leaky, abstraction. They provide value through a mechanism that is cumbersome and doesn't even work well compared to basically all competing solution used outside of Python.
* * *
Anyway, I've used Python for long enough and I've seem many variations of your argument often enough that I'm just... bored. Python packaging is a cluster** and it has sooo many braindead ideas (like setup.py having arbitrary code in it :-| ) that yes, with a ton of work invested in it, if you squint enough, it basically works.
But that doesn't excuse the ton of bad design around it.
> Again, there's a reason basically no other programming language ecosystem needs them.
You have failed to explain how they are meaningfully different from what other programming language ecosystems use to create isolated environments, such that I should actually care about the difference.
Again: activating the venv is not in any way necessary to use it. The only relevant components are some empty folders, `pyvenv.cfg` and a symlink to the Python executable (Windows' nonsense notwithstanding).
> 20 years ago (or whenever they were created), where they didn't even bother to do their homework and see what other ecosystems did to solve that specific problem.
Feel free to state your knowledge of what other ecosystems did to solve those problems at the time, and explain what is substantively different from what a venv does and why it's better to do it that way.
No, a .jar is not comparable, because either it vendors its dependencies or you have to explain what classpath to use. And good luck if you want to have multiple versions of Java installed and have the .jar use the right one.
> They're a bad, leaky, abstraction. They provide value through a mechanism that is cumbersome and doesn't even work well compared to basically all competing solution used outside of Python.
You have failed to demonstrate this, and it does not match my experience.
> it has sooo many braindead ideas (like setup.py having arbitrary code in it :-| )
You know that setup.py is not required for packaging pure-Python projects, and hasn't been for many years? And that it only appears in a source distribution and not in wheels? And that in other ecosystems where packages contain arbitrary code in arbitrary foreign languages that needs to be built on the end user's machine, the package also includes arbitrary code that gets run at install time in order to orchestrate that build process? (For that matter, Linux system packages do this; see e.g. https://askubuntu.com/questions/62534 .)
Yes, using arbitrary code to specify metadata was braindead. Which is why pyproject.toml exists. And do keep in mind that the old way was conceived of in a fundamentally different era.
> And do keep in mind that the old way was conceived of in a fundamentally different era.
Maven first appeared in 2004 (and took the Java world by storm, it was widely adopted within a few years). Not studying prior art seems to happen a lot in our field.
> Feel free to state your knowledge of what other ecosystems did to solve those problems at the time, and explain what is substantively different from what a venv does and why it's better to do it that way.
Maven leverages the Java CLASSPATH to avoid them entirely.
There is a single, per-user, shared repository.
So every dependency is stored only once.
The local repository is actually more or less an incomplete clone of the remote repository, which makes the remote repository really easy to navigate with basic tools (the remote repo can be an Apache hosted anywhere, basically).
The repository is name spaced and things are very neatly grouped up by multiple levels (groupId, artifactId, version, type).
When you build or run something through Maven, you need:
1. The correct JAVA in your PATH.
2. A pom.xml config file in your folder (yes, Maven is THAT old, from back when XML was cool).
That's it.
You don't need to activate anything, ever.
You don't need to care about locations of whatamajigs in project folders or temp folders or whatever.
You don't need symlinks.
One of the many Maven packaging plugins spits out the correct package format you need for your platform.
Maven does The Right ThingTM, composing the correct CLASSPATH for your specific project/folder.
There is NO concept of a "virtual env", because ALL envs, by default, are "virtual". They're all compartmentalized by default. Nobody's stepping on anyone else's toes.
You take that plus Java's speed, so no need for slightly faster native dependencies (except in very rare cases), and installing or building a Maven project you've never seen in your life is trivial (unless the authors went to great lenghts to avoid that for some weird reason).
Now THAT's design.
Python has a hodge-podge of a programming language ecosystem with a brilliant beginner-friendly programming language syntax UX (most languages in the future will basically look like Python's pseudo-pseudocode), that's slowly starting to look like something that's actually been designed as a programming ecosystem. Similar story to Javascript, actually.
Anyway, this was more of a rant. I know Python is fixing these sins of its infancy.
I'm happy it's doing that because it's making my life a bit more liveable.
<C dependencies>
You'd realize why I wrote that if you used Java/Maven. Java is by and large self-contained. Stuff like Python, Ruby, PHP, Javascript[1] etc, are not, they depend on system libraries.
So when you install something on Solaris, FreeBSD, MacOS, Windows, well, then you have to deal with the whole mess.
1. Is the C dependency installed on the system? 2. Is it the correct major version or minor version? 3. Has it been compiled with the correct flags or whatnot? 4. If it's not on the system, can the programming language specific package manager pull a binary from a repo for my OS-arch combo? 5. If there's no binary, can the programming language specific package manager pull the sources and compile them for my OS-arch combo?
All of those steps can and do fail, take time, and sometimes you have to handle them yourself because of bugs.
Java is fast enough that almost everything can be written in Java, so 99% of the libraries you use only have 1 artifact: the universal jar, and that's available in Maven repos. No faffing around with wheels or whatnot, or worse, with actual system dependencies that are implicitly (or explicitly) required by dependencies written in the higher level programming language.
<Virtual envs>
I won't even bother to address in detail the insanity that you described about virtual envs, it's just Stockholm syndrome. Almost every other programming language does just fine without venvs. Also I don't really buy that issue with the lack of portability, it's just a bunch of bad design decision early on in Python's design. Even for Windows there are better possibilities (symlinks are feasible, you just need admin access).
I say this as someone who's used basically all mainstream programming language over the years.
The sane way to do virtual envs is to have them be just... folders. No absolute paths, no activation, just have a folder and a configuration file. The launcher automatically detects a configuration file with the default name and the configuration file in turn points the launcher and Python to use the stuff in the folder.
Deployment then becomes... drumroll just copying the folder to another machine (usually zipping/unzipping it first).
* * *
[1] Javascript is a bit of a special case but it's still slower than Java, on average.