If there is ever a Python 4, the number one goal of the project should be to make Python completely hermetic, repeatable, and redistributable. And there should only be one way to do it.
pip, requirements.txt, and venv are madness. Look at how much suffering they cause! I was dealing with (different) pip issues today myself, and it's not an infrequent occurrence.
I've always found it odd that Python has no built-in version awareness. All it knows is a single installation and a PYTHONPATH, and you fake per-project dependencies by copying everything into a virtualenv. It seems very much bolted-on. (Granted, most other languages don't do it much better.)
I wish you could just install all packages (in multiple versions!) into one place like /usr/lib/pythoncache, and the module loader would then decide which package version to load at import time. You could define your dependencies as usual in reqirements.txt or pyproject.yaml. And you could use either pip or the system package manager to manage the global package cache.
I am 100% with you. Love python the language, but the ecosystem around it was never thought out properly. It grew organically over the years and it is a complete and utter mess.
My last attempt at using pip this very morning to install a bunch of deps crashed and burn with an obscure python stack dump.
Oh, and, in your taxonomy of partial and inadequate solutions, you forgot to mention anaconda, possibly the most invasive package management solution I have ever seen, which exhibits the same problems pip does, i.e. where each install is a total crapshoot.
I started to use python about 20 years ago, which was very pleasant (there was a nearly seamless transition from the old Numeric to Numpy at the time). I was using it less regularly for a number of years and came back to it to find what appears like an absolute mess.
Is there any good overview which packaging solution has which properties, and will work in specific situations?
In fairness, many of these are open source and at least in the case of pip, for many years there was literally only two maintainers although now there a couple more. This is what I learned at one of the python conferences.
It’s odd how open source packages that big companies use can be reliant on so few people.
Edit: although checking the pip github, there more people committing so maybe I got something wrong, but there is far more activity starting around 2019.
> pip, requirements.txt, and venv are madness. Look at how much suffering they cause!
Perhaps I'm just lucky but I never experienced any major issue with pip+venv. It always worked as advertised: create venv, activate venv, install/update pip locally, install packages listed in requirements.txt, and you're all set.
If you write a package and you use pip to install and test it, pip will install the requirements according to what is requested, by default the latest acceptable version.
If somebody else installs your package at a later time, pip will again install the latest fitting versions.
This install will be different from your install, or from other installs other people made in the meantime.
If a newer version if a dependency breaks your package because it has a backwards-incompatible change, your package won't work - and it will not show up in your testing because pip sees the existing, already installed packages and will think they are new enough. So, you'll find "it works for me".
A good examples are packages like python-opencv which break on python2 because the versioning implies they are backwards-compatible but they (or their given dependencies) use syntax which is not supported by python2.
And these things tend to snowball quickly because the number of packages which other packages depend on tends to grow exponentially, without a real upper bound.
And while there is lots of annoyment and cursing about package managers, I think a huge part of the problem is a cultural issue in the python community because people accept and create libraries with backwards-incompatible changes without marking that in the version numbers.
You can go along way with that setup solo, and even in big companies, but it’s not hermetic, it’s not repeatable (byte-for-byte), and it’s not easily distributable.
Assuming we want something that works now and prefer something that works on POSIX systems over something that is not portable at all, would using GNU Guix (as a package manager) be able to improve the situation? As far as I understand, it is hermetic, deterministically repeatable, and redistributable. Just limited to POSIX.
I guess that you do not see conda as a sufficient solution. How would you include and manage binary extension modules written in C which in turn rely on C libraries which might or might not be system libraries? Is it possible to manage that well without cooperation with the host system?
And, as we are at talking about the future, how would you manage native extension modules which are written in Rust?
I'm surprised. I've been using it for ~10 years now, on multiple Linux (CentOS, Fedora, Ubuntu, Debian) and Win7 in the past, and it's always been rock solid and extremely reliable, albeit slower than I' d like (but not too slow as to be unusuable).
I've also collaborated and discussed with hundreds of other conda users, and except for speed and the lack of official lockfiles[0], there were no complaints, only praise.
[0] The functionality was always available, but not straightforward and not named as such.
And pipenv is getting better. Recently thete has been a push of improvement. It still has its issues though. Pipenv or Poetry? Time will tell. Poetry needs to parallelize.
The original author has a habit of abandoning software projects after a while. But I think they managed to get it into the pypa docs (python packaging 'authority'), and so I guess someone has brought it back.
I believe, after some conflict with the community, the original author divested himself of all his F/OSS projects. This resulted in pipenv being donated to the pypa GitHub org. Since then, the project appears to be seeing sporadic development through community contributions.
For context, the root cause to this is actually in Debian, not setuptools (or pip). Debian vendor-patches the stdlib distutils, which setuptools used to depend on, but lost in v50 since it does not depend on the stdlib distutils anymore.
This issue was known for quite a while, and IIRC more recent Debian (and thus Ubuntu) versions contain a fix. But the change was not backported for other reasons I am not personally familar with.
While we are at it, can we finally get rid of python 2? And find some sane way to not need python3.6 python3.7 and python3.8 laying around and installing each their own libs versions?
A few days ago, I learned that the Python interpreter does not specifically uses semantic versioning. Which means that a change in the minor version number (like 3.8 to 3.9) could break backward compatibility.
I don't understand sorry, I must be misunderstanding you. If setuptools doesn't depend on the stdlib distutils provided by Debian anymore, why is this a problem due to Debian? How can you have a problem due to something you don't depend on?
Basically it seems the Debian python does not add site-packages to its search path, but manually installing pip (via get-pip.py) installs things there. And this all worked before, because Debian shipped a patched distutils, but now setuptools vendors that.
So it only affects non-distro-provided manual pip installations? If I use Debian's Python (or some manually downloaded Python) to create a venv and install the new pip in there, it would be fine, right?
That would make it a complete non-issue for me. The global Python installation is managed by Debian. Pip is only ever used inside a venv.
That’s correct. Things only break if you pip install setuptools into the Debian-provided Python globally, not if you use both system-provided setuptools and pip, nor if you pip install setuptools into a venv.
IMO neither party really did anything wrong, the situation is caused by users depending on implementation detail that used to work but no longer does. I would’t say it’s a non-issue though. There are so many instructions out there that recommend `sudo pip install -U pip setuptools`, and Python packaging developers take all the blame when the spacebar no longer overheats the CPU. This situation is, in some way, a microcosm of how people get the impression that “Python packaging is a mess”.
> IMO neither party really did anything wrong, the situation is caused by users depending on implementation detail that used to work but no longer does.
Slightly exaggeratting, but in the same way, you could say "the designers of the C language did nothing wrong with creating that much undefined behavior, any competent programmer just needs to know what they can do and what not".
I am not blaming the developers of setuptools or pip which have very limited resources. But the solution they created so far is just not good enough, and its technical properties contribute to these problems. And other language environments clearly do it better.
Another aspect is that the community accepts libraries which break backward compatibility. This will become an ever increasing problem as soon as it becomes more frequent that programs rely on different libraries which in turn happen to have indirect dependencies which require conflicting versions of the same thing. There is no way to resolve that in Python. It would be much better that libraries which break backward compatibility would get a different name altogether each time, because in this case you could include the different libraries in the same program.
But Python's library authors don't, and this is probably because they want people to keep using their library by given the mere impression that it is compatible, while it is not. And this is just not honest communication. If you break compatibility, you should tell the users of your library. Or better even you don't break it in the first place.
The problem is with the instructions then. So it's an issue in practice only because novices are given bad instructions, without the proper context, that tell them to do things that are known to be a very bad idea.
Similarly, most software used to tell you in the README file to do a "make && make install", sometimes installing into /usr/local but not always. If a user followed these instructions and overwrote half of their distro-provided /usr/lib, things would break in the same way for the same reasons.
The solution is to not do that. Don't mix distro-provided files with things installed from other sources. (As for the problem of people providing bad instructions, I don't know the solution. Perhaps remind novices that all instructions come with an implicit context and may not apply fully to their specific case?)
Pip is already seriously broken, it's just that the breakage has hit one of the main targets for the project.
Part of my day job is maintaining Python for an embedded OS. For self-evident reasons, software for the embedded OS needs to be cross compiled, and Python makes it impossible to cross-compile pip. Explicitly. Upstream's reaction is "you can't cross compile pip, you're on your own." To make matters worse, pip doesn't support non-native Python once it is built.
Of course, this is just the tip of the Python portability nightmare. Python is great and completely portable as long as your target is Linux running on x86_64 or aarch64. It has second-class support for Mac OS and Windows and there are private port projects for a few other system but they generally require heavy patching. For example, the Python code violates POSIX standards in favour of assuming that everything is Linux (hey guys, time_t is not a signed integer that can be used for arbitrary arithmetic!). Executing a whole lot of system code between fork() and exec() is always broken, although it appears to mostly work on Linux as long as you disable certain kernel features, and guess which code base relies on that?
Yeah. Pip is broken and now it's showing up on Debian too. Maybe it'll just get better by itself?
Immutable Infrastructure as Code. Docker containers are the modern gold standard. You can get fairly far on a stock OS with a combination of vendored binaries/libraries and virtualenv/pip, but containers are much simpler.
I dunno if your plan is containers all the way down but for us it was literally docker-compose that was broken.
More important than immutable containers is repeatable builds. I need to be able to go back to a version in 12 months and build the exact same output avec a few changes. Containers don’t help with that.
I am quite fed up with people thinking: "Oh, I'll use docker compose, it'll ve simpler." adding another layerof abstraction on top of something that does not need another layer and the having issues, but not documenting their precise docker run commmands, so that I need to reverse engineer them from the docker compose yaml files.
Literally any change in your system can break. It matters what you do once it breaks, and what I describe makes the whole thing much more repeatable and easier to recover from.
Docker containers builds are extremely repeatable. You run some software called Artifactory, which keeps the built versions of your artifacts, including base image tags. You store your applications in there, as well as the Docker containers that you built. If you need to go back a version, you pull it out of Artifactory. This is industry standard at this point.
> Immutable Infrastructure as Code. Docker containers are the modern gold standard.
Your comment makes no sense once you account for the fact that, as part of containerization security best practices, containers are continuously rebuilt to use the latest version of their base images and dependencies, and those images are then pushed to overwrite the latest release.
Uh, no, you don't ever overwrite a release. You build new versions, test them, deploy them, and if you find a bug along the way, you roll back. You can pin versions of base images and update them when they have security patches, or you can rebuild a base yourself and patch it yourself.
This isn't containerization best practice, this is just plain old Ops best practice that we've been doing for decades. The new part is doing it with containers and building immutable images rather than continuously muting state.
> Uh, no, you don't ever overwrite a release. You build new versions, test them, deploy them, and if you find a bug along the way, you roll back.
No, not really. You pull from image:latest, or debian, or debian:stable, or debian:stretch, or debian:stretch-slim. That's what you use as base images. The reason you do that is security. You need to incorporate security updates in your Docker images, and in order to ensure your images always incorporate the latest bug fixes and security updates you simply pull from the default base images expecting them to be continuously updated.
You absolutely should not ship updates by pulling from an unversioned tag; it's bad for a number of reasons. There's some simple ways to avoid it:
# In your CI pipeline, something like this:
$ cat Dockerfile
ARG release_tag
FROM baseimg:${release_tag}
# etc; i think the container metadata keeps the base version, or you can store it in the container with a RUN line
$ base_version_tag=$(./get-latest-versioned-tag.sh some-docker-repo:my_prefix/my_image:stable)
$ new_container_version=$(./make-new-version.sh)
$ docker build -t my_img:${new_container_version} --build-arg release_tag=${base_version_tag}
Then you run your tests on the new image, maybe do a gradual deployment, and can quickly roll back to the last base image or skip broken versions. Your own container version is always rolling-forward, and you can revert the base image version at any time.
If you do this with a specific stable tree (like debian:8-stable or something) you should only get security updates, but if you use :latest, that's a total nightmare.
I can't edit my old comment, but it's extremely disturbing that I got downvoted so much. What I described is literally the best practice.
If you don't use Docker containers, use some other form of Immutable Infrastructure like Packer VM images. But those are huge and clunky, so you should really get used to Docker containers. You could instead make Linux packages, but they don't make for as Immutable a system as the OS state constantly changes, and packaging is more work in general.
I remember there was time when pip3 used to break apt... or was it the other way around? Or maybe it was pipenv? Whatever that was, it was pretty baffling to see python's stacktrace of an ImportError when you tried to run "apt whatever".
While I'll be first in line to criticize the overall Python packaging-story, you also have to consider how Python itself is being used to implement the underlying platform which the OS's package-manager depends on.
If you allow the pip to make changes to things already present on your system, it make break expectations other parts of your OS may have.
And especially given Python's messy package-management story, combined with the less-than-ideal Python2 to 3 compatibility-story, transitioning a full OS and platform from Python 2 to Python 3 without breaking stuff is no small task.
I'm amazed it even works, and that's before you allow users to mess around with packages using pip.
I think the maintainers thought this is only Debian/Ubuntu problem, but in reality, it affects a lot of other systems. Fedora, Conda distributions seem to have been affected too.
The first thing I do (or rather, ansible does) on any new PC or server is install Miniconda to a location inside my home folder, and set it as the first item in my PATH.
I’ve been down most of the python installation and packaging rabbit holes, and this practice has served me well.
Yeah it always did squick me out that it needed to do that, while also having a special binary manage all the commands. Explicitly, why is 'conda' in my path, but then I need to run 'conda init $shell' to make it really work. It should be handling that internally all by itself, without any help...
IMO, either use system python and system python packages (`apt install python-opencv`), or if you need pip packages that aren’t debian packaged, use pyenv python which takes care of all the pip prefixes. Mixing the two has never gone well for me.
That's not always possible. If you install a package from the system repository that depends on a bunch of python packages, the dependency manager will only be satisfied if they're installed via the repository, since it can't see your pip packages.
So you might need the latest python-opencv from pypi for work, and some desktop camera app needs the one from apt to not have broken dependencies.
You're not going to run into this issue though. Apt-installed python packages will see each other and pip-installed venv packages will live in a separate area. They don't conflict.
You can create virtualenvs that can "see" system-installed Python modules, so this strategy does in fact work. System modules can be used as dependencies for non-system applications with this.
> In short, packaged Debian/ubuntu Python installs things into dist-packages. site-packages is left alone in-case you like to build your own upstream python and install it. Debuntu overrides the distutils it ships to redirect installs to dist-packages.
It gets blurry when you install pip from upstream. But it would call setuptools, which would use the patched distutils so things worked.
>
> Now setuptools vendors distutils this and always installs things into site-packages, which isn't on the packaged python interpreter path, so it can't find modules.
Now, who thinks that vendoring distutils is a good idea? And installing into the base system?
There are things I can't uninstall or install because some needs python2 and others python3 and uninstall python2 or python3 removes one or the other (notably calibre, and I just read the author won't move to py3 and that he can maintain py2 anyway).
For instance I can't install mycli without removing calibre.
Of course it's most likely possible to get around that with virtualenv or whatever is used at the moment but that's a huge cognitive load just to run some apps.
The current paradigm feels OK to me. Unfortunately there are a lot of old crappy "intro to Python" blog posts and tutorials still telling people to use 'sudo pip install' to install packages on their system Python. pip/setuptools do not support that workflow. I think it's slowly getting more widely recognized.
Python -m venv in your project directory. Then source ./bin/activate. Pip install from the project directory as your regular user (no sudo). You can also set up venv (virtual environments) in home or somewhere else instead of for each project. This is called python virtual environments and very easy to adopt.
> Python -m venv in your project directory. Then source ./bin/activate.
And how do I do that from inside an already running Emacs session? (Or Vscode?)
Needing to "prepare" an environment before you can work with it, prevents you from simply "jumping" into a task from an existing workflow.
Working with .NET, Rust, Node, go, Haskell, etc... I have no such issues.
This "venv"-requirement only applies to Python, and it's really the opposite of ergonomic. I really think it's about time the Python-community starts recognizing this.
I generally recommend against `source ./bin/activate` for virtualenvs — it's convenient if you're using a single shell session interactively, but it plays quite poorly with the Unix process model of environment variables being inherited from parent processes to child processes.
Instead, I recommend just running the other executables in the bin directory. `./bin/python` will drop you into a Python REPL with all the packages installed in that virtualenv available. Your IDE should let you specify what Python interpreter to use for running tests and other integrations; give it the path to that Python executable and everything should Just Work. Likewise, if your package uses setuptools entrypoints, they'll be installed into the bin directory too, and you can just run them with no further configuration.
For what it's worth, Rust (for one) works pretty much the same way. The only difference is that Rust doesn't have a concept of dependencies installed system-wide, they're always handled per-project like venvs. As a result, they don't have decades of stale tutorials online telling people to do the wrong thing.
I usually give myself a `./exec` shell script that sources `./bin/activate` then `exec $SHELL $@`. That gives me a subshell with everything set up, so when I want out it's just Ctrl-D.
Now that I'm used to the subshell idea, I use the same concept across ruby, node, and python; it's almost always a better fit than what the usual packaging tools give you. I don't need `bundle exec` in ruby, for instance.
> Working with .NET, Rust, Node, go, Haskell, etc... I have no such issues.
Respective solutions: NuGet, cargo, npm, go get, cabal. They all solve pretty much the same issue as venv and prepare your project in pretty much the same way. Raw venv can be a bit rough, but poetry solves that.
> And how do I do that from inside an already running Emacs session? (Or Vscode?)
From VSCode I just use the integrated terminal to create the venv, vscode's default python extensions recognizes in-process venvs by default and uses them. (The way it installs tooling is broken for them, though, but that's a bug in the extension.)
Though it would be nice if Code integrated poetry so that just having a poetry-compatible pyproject.toml was sufficient.
> Working with .NET, Rust, Node, go, Haskell, etc... I have no such issues.
For most of those (Haskell with Cabal included), they have the same basic issue and solve it with a project file which is pretty similar in purpose and effect to poetry’s use of pyproject.toml.
Ruby has this figured out well by comparison. Install the bundler gem (which you're going to be installing anyway to get dependencies), cd into the project, and prefix your commands with `bundle exec` to use project gems
Remembering to start/stop virtualenv and remembering the names of your virtual envs is one too many steps
Or just use poetry/pipenv shell (or poetry/pipenv run for one-off commands). Both provide a good wrapper, although poetry seems a bit more stable these days.
Use your OS to install the stock Python and the stock setuptools, wheel, and virtualenv Python packages. Then make a virtualenv for every single Python application (virtualenv ./some app/). Then make a requirements.txt file with the pip packages you need and their version numbers and install with ./someapp/bin/pip install -r requirements.txt. If you do something bad, rm -r ./someapp/ and start over.
I disagree to use the stock OS for any part of Python you're using as a regular interactive user. The easiest and safest way is to use a Python vendoring tool like pyenv to install a Python version of whatever you need. After that you can use regular pip and virtualenv without any special paths, and the environment is local to the user, so you're unlikely to interfere with another user's environment.
Pyenv is not unlike nvm for Node. Python moves too fast to expect the OS distributions to keep up.
I've been a regular Python user for a decade and every Python app I've ever used has worked on stock Python. And vendor builds often ship with patches that make it work better on the OS in question. But I can see how devs often want bleeding edge releases.
From a stability, reliability, and CI/CD performance standpoint, I would discourage building your own Python. Things break more the more often your base version changes.
To be fair, any language (and database) moves faster than distributions. I use language package managers or docker for Ruby, Python, PHP, Node, Elixir, PostgreSQL and MySQL. I recommend asdf to support multiple languages and databases in the same project.
It's not necessarily easy to install "stock python". e.g. installing python 3.7 or newer on ubuntu requires quite a bit of initial setup.
You'll be required to install a ton of apt system dependencies, then add a custom system apt package index (ppa), then install it from the new repository and hope you managed to get all required dependencies. I failed several times in a row last time before I got it working, after trying 2 different methods.
You should use `pip install --user`. Or even better, use virtual environments instead. Or, if you need to install system-wide tools with pip, install and use `pipx`.
The current version of pip automatically goes to --user mode if it detects it doens't have permissions to access the system pip dirs aka not being run with sudo
After parsing the sibling threads, in this case there were only 4 distinct answers:
- Python -m venv + ./bin/activate + pip
- poetry
- pipenv
- pip (without isolation of projects)
You may add varying recommandations on the usage of OS python or a user installed one, but I think the same concern exists with most languages.
The main problem is that Python's mantra is supposed to be "There is only one way to do it". Less than two years ago, I read the official PEP doc about "Managing Application Dependencies", and it was a mess. pipenv was recommended (and still is), but there was a critical bug that ended in fatal errors on my development system.
"Python -m venv + ./bin/activate + pip", pipenv and "pip (without isolation of projects)" are really all the same thing, just with different options (or a script around it).
Basically, start with "pip (without isolation of projects)" until it doesn't work for you because of library version clashes, then move to either pipenv or "Python -m venv + ./bin/activate + pip" (which are basically the same thing).
RealSense is "broken" in that they don't provide builds for all the versions of Python their users use. But that's because "Officially, Librealsense is only currently supported up to Ubuntu 18.04 LTS"
Perhaps also your process is "broken" for depending only on pre-compiled builds; the published solution is to build from source. https://github.com/IntelRealSense/librealsense/issues/6296#i... says it's possible. Or "broken" for using an unsupported environment and expecting it will work.
Software in general is broken. It's amazing that it works as well as it does.
I tried that, it didn't work, and they keep closing issues about it instead of addressing it. They are a company with a 200B market cap, and it's already 5 months since the release of 20.04 and they still can't get their drivers straight.
People like me want to use their hardware. I don't have time to be debugging their drivers -- I have other work to do. I don't think it's unreasonable ask for them to support the latest LTS release of the most common Linux distribution in a timely manner with pre-compiled binaries that just work, in return for the $179 that I and tens of thousands of others paid for the device.
This isn't some open source community software that I'd be glad to pitch in some time and help debug. This is a driver for a device that I paid a for-profit company for and I expect timely support especially for an exceedingly common OS that I use.
Sure. But they don't. So is the error in them, for not following semantic versioning, or is the error in you for incorrectly assuming they follow semantic versioning?
Are you really just complaining about the version numbering? Ok, let's say they follow semantic versioning, and instead of calling it 3.8 they called it 4.0.
The package you use would still be broken.
Now what do you do?
Why don't you install Python 3.7 yourself and pip install the pre-built binary?
Python has never used semantic versioning. There is a long history of small, breaking changes between "minor" releases of Python. Eg, Python 3.7 made `async` and `await` global keywords, breaking any case where old code used `async` as a variable name.
I have never really understood how Python dependency management got broken in the first place. The basic model is the same as the basic Unix shell model: you have a path, which is just a list of places where executable thingies can be found, and whatever wants to execute the thingies looks in each place in the list, in order, until it finds the thingie it wants, or it bails out with an error if it's gone through the entire list and can't find what it's looking for. In a Unix shell, the list is $PATH (or maybe $LD_LIBRARY_PATH, since most Python dependencies are libraries); in Python, it's sys.path. Installation of an executable thingie then is simply a matter of "make sure this ends up in one of the places listed in the path". What is so hard about this?
So, my most recent success has been using pyenv and pipenv.
I install pyenv first, then install any specific point releases I need, then switch to the python version I want to use, and then use pipenv to install dependencies or run commands in the virtual environment.
Version numbering and labeling of the packages--the same way Unix solves the same problem with shared libraries. A program that wants a specific version of a library looks for it by specific version.
Honestly, just throw everything in a venv and forget anything about needing to manage platform versions again. Unless, of course, virtualenv can't be installed on the host in the first place due to setuptools/pip issues :)
I assume the poster meant "Debian packaging", but yes. Debian and Ubuntu remove venv support along with distutils from the standard library by default.
This is annoying, but it does encourage the principle of "the system python is for the system, and only incidentally for the users of that system". I'm almost always in a non-system Python for anything interesting, unless I'm writing something to integrate with the OS.
It breaks interrogation of the Python interpreter, since those functions are part of distutils. It also breaks creating virtualenvs if it relies on the venv module.
Yeah, it creates a bootstrap problem that wouldn't exist otherwise. I'm pretty much of a mind that meta-tools like venv shouldn't be written in the language they target, precisely to avoid problems like this.
Well, if it's a standard library function, it should not matter what language it's written in. The _intent_ is to make it accessible to everyone, and Debian breaks that assumption.
This is so true. We should patch setuptools to easily be able to create deb/rpm in addition to wheels so that open-source projects could easily host PPAs of packages.
the reality is that there are binary needs at the OS link level, and then there is pure-python or ML with drivers or whatnot, per user.. its not actually easy to divide which is which ..
apt is completely inappropriate unless you're using 3 year old dependencies. Ubuntu 20.04 (released in April) doesn't have Python 3.8, which was released in October 2019.
This is patently false; the default python3 on Ubuntu 20.04[1] is 3.8.2. A separate package python3.8 is also available in built-in repositories for all supported versions of Ubuntu going back all the way to 18.04[2].
pip, requirements.txt, and venv are madness. Look at how much suffering they cause! I was dealing with (different) pip issues today myself, and it's not an infrequent occurrence.
Rust's Cargo is a good reference point.