Download it and see if it will run on your OS. I'm getting the standard prompt on every OS so far. On Windows you just type python.com. On Mac OS X I had to chmod +x, allow it in security settings, and use bash -c './hello.com' to execute it. Simple chmod +x python.com; ./python.com on Linux.
Does this mean I can build portable Python applications like I do with Go without having to worry about the glibc version on the running machine in case I had used CGO?
Cosmopolitan Libc allows you to build programs that run on six operating systems (Linux, Mac, Windows, FreeBSD, NetBSD, OpenBSD). I ported Python 2.7.18 and 3.6.14 to build on Cosmopolitan Libc because I wanted to have the same Python executable and my scripts with me everywhere :) I've updated the blog post for context.
The flask demo is unnaturally slow because my old computer is slow (running python tests in the background). plus I was using `_dummy_thread` Python3.6's pure-python threads implementation.
APE is for αcτµαlly pδrταblε εxεcµταblε [0], a kind of executable bundle that runs from the exact same file on Linux, MacOS, Windows, DOS, BSD, etc.
Cosmopolitan is a libc for that bundle [1], so that you can write your C code just the once, and then you have just the one output executable. No download-per-platform, or that's the goal.
It's a nitpick but I wish everyone stopped spelling it like that and call it Actually Portable Executable at least. That kind of spelling really messes up with my reading comprehension, and I think it does for many people.
No inside knowledge here but I figured Justine intends it as a shibboleth.
I use a magic string somewhere, which has control codes, three ASCII digits for versioning, and an RTL UTF-8 character, and is exactly eight bytes wide.
Anything munges my file, I'll know. Meanwhile I can just mask it.
> No inside knowledge here but I figured Justine intends it as a shibboleth.
I would not call it a shibboleth, but vague, generic terms are known to be for most purposes unsearchable, and having a really unique string identifier for your thing could make it more valuable.
Obviously we're from different tribes because clearly you're wrong, and I'm compelled to cut your throat and throw you in the Jordan river. I think. Been awhile since I've reviewed the source material.
My screen reader doesn't pronounce it the way you'd expect, because it's not using Greek characters the way they should be, but it is the name. Transliterating it, is as wrong as using the original, but the author prefers the original, so that one wins out.
Screen readers can surely handle Greek just fine, the only problem with the given spelling is that it uses "lookalike" glyphs instead of corresponding characters from the Greek alphabet (eg. use of "mu"/µ for "u").
Of course, I am sure some screen readers will simply take a custom text preprocessor that will turn l337sp34k and similar spellings like this one into regular text, but I am certain none of them have it as the default.
Well, αcτµαlly pδrταblε εxεcµταblε is what the author calls it. I appreciate it when people respect author's choice of naming. If I had a wish, it would be for the author to change the name.
The Greek delta being used as a "o" irks me a bit too, because to me it looks very much like a consonant, but I make a one-off exception for this project.
SEO has become such a tainted word that clearly it cannot be used anymore to shorten "Making something useful easy to find in search engines by anyone who might be interested" - people think anything mentioning SEO is from narcissists or scam artists, and downvote.
What other term would people suggest then? Surely it's a real concern for projects to be easily accessible by people looking for them.
DOS .COM programs were pure maching language programs with no OS-specifics like headers or syscalls (though running in real mode, they did expect BIOS interrupts), so that makes sense to me: they are trivially portable.
I say that because "python.com" looks like a URL. DOS was a loooong time ago (I used it in my youth) and well out of the memory of most internet users today. So instead they see "python.com" and think it's a web address. And while I thought it would direct you to a website about python, you know where that address goes? To a website that has something to do with creating your own camsite. Yeah... that kind of camsite.
And then, it's bad because a user trying to search for it using the universal search/URL bar in their browser can't just type that in and hit enter to search for it, they have to manually select that they want to search for it, or else they go to the site about starting your own camsite.
AND THEN... once they have googled for it... none of the results on the first page have anything to do with this program as far as I can tell.
So yeah, weird, obscure, confusing branding that makes the program hard to search for.
EDIT: I could be missing something? Does this program have a different name and python.com is solely the file name of the executable or whatever? It still seems like it could be more descriptively named.
Yeah, I was aware most would take it as a domain name, but we _are_ on HackerNews, and I think .COM is still an applicable executable extension on Windows. As such, it strikes me as a decent name (when looked at independently).
But ouch, I haven't looked where python.com might be pointing to today, so that's indeed unfortunate.
As for search results, I think the most obvious candidate for bad naming is... the Go language. ;-)
Btw, this is part of the Cosmopolitan (referenced multiplatform libc implementation) monorepo, and only lives in the `third_party/python` directory. I am not sure if it's customary to name Cosmopolitan-linked binaries with a ".com" extension (perhaps it also alludes to the name "Cosmopolitan", but I guess then it'd be "cosm" or "cosmo"), but it definitely does not seem to be a public name you'd search for.
Searching for "python.com" (in quotes) on Kagi or Google returns a bunch of Python-related .com domains, whereas searching for "Actually Portable Python" returns the above site as the first hit — that's the name one should be looking under for this. Though I am sure it'd be nice if a more generic "multi-platform Python binary" also matched some of this (though platforms sometimes mean architectures as well).
I was confused by this naming scheme and I am an (allegedly) seasoned software engineer of the DOS era (team 6.11a).
Reading these comments about why it's named python.COM, yes that's pretty clever but it didn't even occur to me when I first read the post and I suspect the majority of people on hackernews had the same impression.
Go is definitely also bad naming, but the domain is golang.org and the canonical search term for the Go programming language is "golang".
In this day and age, we should probably be optimizing for clarity and searchability instead of "clever use of an outdated tech naming" which half of the engineers nowadays wouldn't even know about.
It's named .com because it's a flat executable format. If you map it into memory and start executing at the first byte (MZ which means JG +69) then it'll just work.
If you are looking for a multi platform build of a specific version of python built with an obscure libc for some niche use case it's _very_ likely none of that is an issue for you, at all.
I would say .com extension as an indication of an executable is acceptable exactly because we are on HackerNews (and not W3Schools.com, for instance). Basically, IMHO, "hackers" are supposed to know a bit of (arcane) history as well: "hacking", to me, presumes a deeper investment in the field. Otherwise, they might only be (web) software developers (or engineers).
But to each their own.
(I think the most unfortunate thing is what content is present on the python.com domain, as a sibling comment highlighted)
And how is headerless raw assembly that expects the bios interrupts to be usable "trivially portable"? If it was, it would run on modern amd64 OSs without emulation, but it doesn't.
COM "format" is "trivially" portable for a limited subset of functionality where you don't interface with the rest of the system except possibly memory (I think .o files produced by GCC compiler are very close to pure machine instructions too, before they get massaged into an ELF file with ld): in the sense that you can wrap them in a function and just execute them on the same architecture.
The fact that system interfaces like BIOS interrupts may or may not work is insignificant when you start thinking of it as only a playful name.
And let me repeat a word of warning: this is only my guess, I am unrelated to the project.
python.exe is the only other option to fulfill the Actually Portable requirement on Windows.
Thing is, an APE is a COM file, not an EXE. Different headers, which contemporary Windows no longer cares about; but why lie about it? python.com is in COM format, as far as Windows is concerned.
Added for clarity: 'no longer cares about' meaning, Windows loads executables by checking the header for COM or EXE format and behaving accordingly, it's which of the two valid extensions gets used which it no longer cares about.
Just to clarify, APE is both an EXE and a COM file. The COM file format is basically just flat executable code. The only thing you need to do, to be a legal .com file, is if the program is loaded and execution starts from the first byte -- it works. APE meets that requirement for Linux and bare metal. It uses a polyglot to tell if the processor is running in real mode or long mode within its DOS stub. https://github.com/jart/cosmopolitan/blob/066ed2b2b229dce4d1... APE is also a legal EXE file because it has the MZ header. However... I thought it'd be more cool to signal as a DOS binary than to signal as a Windows binary, since signalling as the latter would probably be a turnoff for the Mac/Linux/BSD crowds, who are also supported by APE.
APE is cool, but other than it being a bar trick I fail to see much usage out of it. Especially in case of an interpreter - like, just have the native version installed on the target OS.
End user applications, sure… but the raw python interpreter? If you want it, you have to install it. If you want to install it, you go to python.org and install it.
This solves… a headache for the maintainers, having to build multiple packages?
Not having it installed is definitely not solved by this.
Bundling python packages into a single binary is definitely not solved by this.
> End user applications, sure… but the raw python interpreter? If you want it, you have to install it. If you want to install it, you go to python.org and install it.
If you want just the raw python interpreter, you're not the end user?
> Not having it installed is definitely not solved by this.
But not having to install it is the point?
> Bundling python packages into a single binary is definitely not solved by this.
It is? There's even a working example in the article?
mkdir -p Lib/site-packages
# specify python version to pip if necessary
/usr/bin/python2 -m pip download flask -t .
# start "build"
cp ./python.com ./my-release.com
printf '%s\n' '-m' >./.args
printf '%s\n' 'my_module_name' >>./.args
# wheels are just ZIPs, use unzip if pip complains
./python.com -m pip install flask*.whl -t ./Lib/site-packages
./zip.com -qr ./my-release.com ./Lib/site-packages/
./zip.com ./my-release.com ./.args
# optional cleanup
rm -rf ./*.whl ./Lib/site-packages/
./my-release.com
You don't understand how the packaging of python applications works if you think the efforts of PyInstaller, py2exe, Nuitka, etc. can be this trivially overcome.
This makes what is already a pain in the ass (packing a python app into a single native distribution) harder to do, for the 'cute' ability to have a single cross platform binary.
...and it is cute. Sure, I get it. ...but yeah, you just zip up all the python dependencies and it's fine? So good. Compile all the native parts or whatever, I'm sure it's easy.
Look, I tell you what, you go and prove that all the smart people who've been trying to solve this for the last 10 years that there's a new good way of doing it, and they (and I) will all be very happy.
Until then, this is a proof of concept for a trivial use case that makes packaging a trivial pure python application somewhat harder to do than it was already.
> It is?
It is not.
Bundling python applications into a native binary is already a badly-solved problem with a bunch of irritating problems and caveats; this doesn't add anything meaningful to those existing efforts or fix any of the issues they have.
Seriously; go and look at the efforts those projects go to, to make what they do, work.
it isn't the solution, but it is a solution. it might be the solution for a particular subset of tools, which is still great, because it usually was a PITA even for those constrained use cases.
The OS-specific interpreter can be bundled with the application itself, and then no need to install anything (though ideally a package manager would do its job and resolve dependencies beforehand, making it moot. This windows-style random exe hunting and downloading should just die already. And it doesn’t have to be “installed”, nix solves this problem by nix-shell -p package “installing” the package temporarily, making it available in the shell only.)
> The OS-specific interpreter can be bundled with the application itself
Why does the interpreter need to be OS-specific at runtime?
If you're "bundled with the application itself", and your application is OS-specific anyways, then you should of course bundle a optimized OS-specific interpreter.
But, for an actually portable application, that runs on several OSes, you probably want an actually portable interpreter?
> And it doesn’t have to be “installed”, nix solves this problem by nix-shell -p package “installing” the package temporarily, making it available in the shell only.)
`nix-shell -p package` is a very bad example of your point.
* It represents a "not bundled"/"not embedded" use case;
* You "just" need to install Nix, then;
* You "just" need to invoke Nix using the correct derivation, then;
* Nix "just" need to temporary install the native interpreter and all other dependencies specified in the derivation, then;
* Nix "just" need to temporary install your packages and all other dependencies specified in the derivation in a temporary location.
all in order to not need to have it to not be installed.
Even as someone that likes Nix, this doesn't make a lot of sense for the sake of argument.
Then you either need a package manager (Windows users without Scoop or Chocolately are out, destroying portability) or you need to provide multiple binaries that the user chooses for different platforms with the interpreter bundled as you said.
But the availability of other options with tradeoffs doesn't answer my question as to why this method is just a bar trick.
A Go binary or similar is an option, but it doesn't make other options mute.
There are definitely real uses for this is this got traction and wide adoption and developed into a robust product - for example, games would just work on Linux or windows without the need for specialized drivers for either, since Nvidia would just write the drivers and compile them with this which would make them effectively multi OS, followed by all game engines targeting these drivers, and so on.
However to wish for that to happen on the same level as wishing that Linux was the dominant OS for personal computers, which would render APE mostly moot except for a few specialized cases.
How do you imagine the drivers would just run on any OS, talking to the kernel in a very very kernel-specific way? This is just a hack in the respective executable file format headers (a genius hack nonetheless from Justine), but it only works for very very basic things. Anything higher level is highly OS-specific.
>This is just a hack in the respective executable file format headers (a genius hack nonetheless from Justine)
Its a lot more then that. Look at the code. https://github.com/jart/cosmopolitan
You compile an executable with flags --nostdlib and --nostdinc because it includes cross platform standard libraries mapped to the OS specific magic numbers for system calls.
Think about a desktop computer with a graphics card. You can install either windows or linux on it. In both cases stuff gets rendered to the screen using the graphics cards. That is done through the driver, which maps higher level commands in libraries to instructions that graphics cards understands. While library format, is OS specific, the driver is essentially doing either ioctls or memory mapped io, both of which are the same x64 instructions.
So compiling everything with APE, means that library entry points are platform agnostic, and everything up the chain all the way to the game engine.
And, yes cosmopolitan only works on very basic things, as would be expected by the small number of people developing it, thats why I said IF it was adopted widely and developed further it would be possible.
> That is done through the driver, which maps higher level commands in libraries to instructions that graphics cards understands
Which is a very highly privileged thing to do, that you can’t just circumvent. No mainstream OS does userspace drivers.
And honestly, now you just introduce two levels of abstraction - one at the APE level, and one at Python’s level, so I don’t get that.
Also, that’s why we have cross-platform libraries. The executable-header is hardly the “bottleneck” in cross-platform tooling - that’s why I think it is hardly more than a very cool and genius hack.
AFAIK the idea behind 'actually portable executables' only works for POSIX-style command line programs. As soon as system-level DLLs (like OpenGL/X11/XGL or D3D11/DXGI) need to be accessed the problem becomes a lot more complicated (solving this problem would be massive though).
Follow the Cosmopolitan Libc project (also consider supporting and/or contributing to it), these things are on track to happen soon :)
TLS will probably happen first -- Cosmopolitan already has mbedTLS for redbean, so a modified `_ssl.c` should get us most of the way there.
Recent Python versions ... what version is Python on right now? 3.10? (3.11 apparently) I picked 3.6 at the time because Cosmopolitan Libc did not have threads then (it does now, pthreads API is being filled as of this writing), and also because it wouldn't have too much churn. Perhaps after 3.11 gets stable I'll port it to use Cosmopolitan Libc.
Here's the latest update on TLS https://github.com/jart/cosmopolitan/blob/master/libc/runtim... We need to rewrite the x86 binary code at runtime in order to support it on Mac/Windows. I managed to speed it up from 30ms down to 1ms of startup latency for python.com!
When people complain about Python distribution, having to download different binaries for different operating systems is almost never part of the complaint. So, the problem has been “solved” by pyinstaller and co. ages ago.
Any problems still haunting pyinstaller and co. at this point will also show up for APEs, and solving them when you only have a lowest common denominator toolset at disposal will be harder.
I got interested in APEs because I got lost in DLL hell a few times with PyInstaller lol. IMO PyInstaller is pretty cool, but a bit overkill sometimes. At present I find APE Python a perfect match for one big use case (a bunch of my thrown-together shell utilities I can take and use everywhere).
Like after I wrote this blog post, I wrote a small Python app using Flask/SQLite/D3.js, put everything in a single file, just copied it onto computers with Debian/Fedora/FreeBSD/Win10, and now I just access the info I need on those systems via the browser. Providing a browser GUI makes it easier for my colleagues as well, some of whom don't use terminals at all :)
For C extensions (where I've stubbed my toes quite a bit with PyInstaller), I think a nice goal with APE Python would be to have a website like Christoph Gohlke's (https://www.lfd.uci.edu/~gohlke/pythonlibs/), where you just select the packages you would like, and you download them compiled as a single-file executable you can use anywhere. This is an interesting problem, and there's been some work towards solving it, let's see if we can have an elegant solution.
> When people complain about Python distribution, having to download different binaries for different operating systems is almost never part of the complaint.
Have you done a study?
This reminds me of when I've seen people say things like, "X is a showstopper for me when it comes to C++ [or whatever]", and someone responds, "That's not a problem in practice. People who write C++ don't care about X". And it's like, "Well, yeah, that's the nature of 'showstoppers'—all the people who do care about X noped out a long time ago." It's an intellectually dishonest way to justify choosing not to confront the X problem.
If I understand it correctly this is only a distribution of Python itself, not third party libraries or user code. So it only solves an insignificant part of Python's distribution issues.
One of us is confused, and it very well might be me and GP, but I thought that Go can cross-compile to other platforms (different OS and arch combinations), but cannot produce (at least by default?) a single binary that can run on multiple platforms, like this article does.
Yeah Go does not do that. I don't think any lang does
I think it's pretty much a unique feature of APE, because APE binaries modify their own code after the first run to nativize themselves. (Right? Or do the new versions work different?)
I can't tell if my disdain for this is real (Since I really like having my exes read-only and having a consistent hash) or just sublimated envy (Since most desktops will in practice have every file marked as read-write, and if it works it's not stupid)
I like the idea of combining this with pyScript to create portable “GUI” apps. Use this for a local server, open the users browser and point it at it. Websockets/pickle to bridge to a UI written in pyScript.
Does cosmopolitan have an API to open the users default browser? I know Python itself does…
Nice idea! IIRC redbean has the `LaunchBrowser` function (https://redbean.dev/2.0.html#LaunchBrowser) to open the user's default browser, and the `cosmo` module in Python is exactly for providing Cosmopolitan Libc goodies via a Python API. You can probably copy some of the LaunchBrowser code into `third_party/python/Python/cosmomodule.c` and implement the feature you're looking for. Submit a PR :)
I'm truely impressed. I don't know if this will become useful in real software, because you'll likely have dependencies (which in turn depend on OS features) pretty fast. But nevertheless, one of the coolest projects out there!
I'm thinking this python.com file would be so, so useful for DevOs / admin tasks. Use it as a lightweight binary to run Python scripts for all platforms. Would probably be very useful for setup / installation scripts or install commands.
Yup, this is one big reason why I wanted Python to be Actually Portable -- I have a bunch of tiny scripts that are slightly complex for sh (some of them were Windows CMD .BAT files lol) which I like to use everywhere. By building Python with Cosmopolitan Libc, I can take a single binary and my scripts with me! I just add my scripts into the APE ZIP store via `zip -r ./python.com ./.python/*.py` and it just works. I only have VMs now, but a while ago I had Debian 10/Fedora 35/Windows 10/FreeBSD 13 all on the same network, and it was super convenient to have the same script/executable work everywhere.
The author compiled the Python interpreter as an Actually Portable Executable (APE) file.
APE files (usually using a `.com` extension) are single executable files that can be run natively on many major operating systems. Kind of a "polyglot" executable file that most major operating systems just understand natively.
The implication is that you can now package Python code as a single file that can be directly run regardless of the operating system the user is on. No installers, no dependencies on what must already be installed, no separate downloads for different operating systems. Just one single file, click and run.
I think what is happening here is that the python interpreter can be used as an executable. This way you don't have to install python on the host machine and can carry your programs on a USB stick for example.
Python is not slow enough. You can now give people a collection of bits that will run python even slower on multiple architectures. As a bonus, you cannot use any of the C packages like numpy or pandas that might cause it to run faster.
Yes. I initially ran into binfmt_misc issues, did a bit of reading on Justine's github repo for Cosmopolitan and found it can be an issue, especially if you have WINE installed. Once I disabled it
I wanted Python to work across at least Windows/Linux because I have a bunch of these tiny scripts that I keep running everywhere. (For example, I always use `python.com -m http.server` when I want to share files in the local network without SSH). Cosmopolitan Libc made it easy for me to run my scripts across Linux/Windows and more, and the running scripts can be accessed via a browser GUI, which makes it very convenient if someone not use to CLI has to monitor what is going on. Plus I can update my scripts any time by just adding them to the executable via a `zip` command.
> Cosmopolitan Libc does come with some tradeoffs as well
> (static compilation, C codebases, no multithreading
> (Update 2022-07-27:, no multithreading yet,
> the pthreads API takes a while to fill
Toy.
> Regarding python packages with C extensions, the build
> process for adding them to the APE is rather unwieldy
> (2022-07-27 adding C extensions to the APE is much more
> elegant now because of the Cosmopolitan monorepo
Horrific malware. Unchecked C (even libsafe) is bad.
Edit: Nevermind that each app using this software will need to be individually updated every time a new security patch for Python comes out.
The downvotes are because you're not really adding anything (I think). This just reads as empty complaining. You have an edited-in point that isn't widely seen as an issue (it is the case, but why do you think it's bad?). That makes it tedious to read, not very stimulating.
This APE format is not "write once run anywhere", because it targets a specific architectrure (64-bit x86) and won't run on other architectures. Just a toy format and I see no use for it.
What we need instead is better support in Linux for legacy Linux and Windows applications so that Linux could run any proprietary software written for it or for Windows (because there are no sources for it and it cannot be recompiled). Today it is not so: for example, very old Firefox builds don't run or crash on modern Linux.
> This APE format is not "write once run anywhere", because it targets a specific architectrure (64-bit x86) and won't run on other architectures.
> What we need instead is better support in Linux for legacy Linux and Windows applications (...) Today it is not so: for example, very old Firefox builds don't run or crash on modern Linux.
But that is the exact reason why "legacy Linux and Windows" doesn't work, and why APE works: you should target the "specific architectrure" directly instead of hopefully trusting that the entire underlying system is a completly static target.
That or using containers/shim layers, those of course bringing such entire underlying system with them.
We should just go back to the good old days and ship custom operating systems with our applications. Then you can just boot to them and be sure the underlying system is what you want.
I agree. Plus it's only relevant for very simple (in terms of API usage) CLI programs. Good luck compiling a Qt app with this for example.
I think WASM is a much more promising avenue for true cross platform executables. You don't have the architecture issue and aren't using weird old formats.
They wouldn't be solved because the problem is that newer versions of shared libraries are not compatible with older applications. There is no problem with ELF or PE formats - ELF is supported on Linux natively, and PE is supported using Wine.
APE format doesn't solve the problem with shared libraries changing over time, it dosn't solve the problem that different plaforms provide different APIs (for example, windowing APIs, audio APIs etc).
Python 3.6.14+ (Actually Portable Python) [GCC 9.2.0] on cosmo
Type "help", "copyright", "credits" or "license" for more information.
>>: print("hello world") hello world
>>: import asyncio Traceback (most recent call last): File "<stdin>", line 1, in <module> ModuleNotFoundError: No module named 'asyncio'
>>: import socket
>>: s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
>>: s.connect(("www.google.com", 80))
>>: s
<socket.socket fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('192.168.21.19', 63468), raddr=('142.250.70.196', 80)>
This work is going to be of interest to a lot of Python devs!
Edit: here's the link to the file: https://justine.lol/ftrace/python.com
Download it and see if it will run on your OS. I'm getting the standard prompt on every OS so far. On Windows you just type python.com. On Mac OS X I had to chmod +x, allow it in security settings, and use bash -c './hello.com' to execute it. Simple chmod +x python.com; ./python.com on Linux.