That would be cool. I've been looking for a way to have to re-download a game every time I play it. And native games just run at way too high an FPS...
If someone is looking for some low hanging fruit in the way of contributing to cosmopolitan, they could teach its scanf()[1] family about floats[2]. (May not be as low-hanging as it first appears though, as converting floats correctly is trickier than it might first appear to be. The comments in the code say it can do floats, but the comments are mistaken.)
Note that this is not Blink the rendering engine that powers Chromium and related browsers, but https://github.com/jart/blink, “a virtual machine that runs x86-64-linux programs on different operating systems and hardware architectures”.
Where this talks of running GUI programs, I presume it means that the VM can act as an X client. You would thus still need an X server. I don’t know what may exist along those lines for the web. As for native/desktop hosts, well, I hope you weren’t ever trying to use emulation as any form of security, because the likes of Xorg really aren’t designed for that use case.
> Do you think the Xorg devs are spreading lies about Xorg?
Some are, yes, this is incontrovertibly true. But this one? You should hit the "history" button that's in the corner of every wiki. That page hasn't been edited at all since 2013, imported from some other document written I don't know when, and literally the only thing it actually says about security is that "needs to be designed with security in mind".
What does that mean? What specific shortcomings have they identified?
The fact that any Xorg client can become a key logger without any user input or authentication is a pretty big security hole imo.
By design Xorg has no isolation between clients so they can all read each others input, control others windows, and inject keystrokes into other applications. That’s unacceptable in the modern age and makes any attempt at sandboxing or separation of privileges for GUI applications completely pointless.
> The fact that any Xorg client can become a key logger without any user input or authentication is a pretty big security hole imo.
This "hole" doesn't exist. For an X client to capture input, it must be authenticated by either the unix user permission or by an access control list (where the default is to deny). Individual clients can also be marked untrusted which sandboxes them to some extent (though not as much as using a separate X server of course).
I'll grant that in practice, most the time these restrictions are very lax... in part because they can break some applications. But at the same time, in practice, it doesn't seem to matter that much since either you're running things you trust anyway or if a malicious application has access to your X connection they also have access to all your other files so you're in trouble anyway.
Apologies for not answering your question directly, but I'm pretty sure this is what XWayland does to allow for compatibility of X apps ontop of wayland.
I imagine it's mostly the issue related to applications having too much access to thinks like the keyboard and other windows without a permission system? That's a large part of what makes Wayland different from X11.
I’m curious how you think it an outright lie. I’m speaking of the consequences of Xorg’s basic lack of isolation (because it wasn’t designed for use cases where that matters): that granting access to the X socket allows you to inspect, intercept and modify inputs and outputs for things like key strokes, window contents (yours or others’) and clipboards.
> I’m speaking of the consequences of Xorg’s basic lack of isolation (because it wasn’t designed for use cases where that matters): that granting access to the X socket allows you to inspect, intercept and modify inputs and outputs for things like key strokes, window contents (yours or others’) and clipboards.
This is not true! Both the protocol and the implementation allow for significant separation of server permissions (for example, notice the difference between ssh -X and ssh -Y), and the peer-to-peer nature of things like clipboard means it is quite easy to deny requests on an application level if they're written for it.
If the X server were running in the VM, how would it display things, or get keyboard and mouse input? You’d need to provide emulated hardware devices, which could be something like /dev/fb0 for output and /dev/input/ devices for keyboard and mouse. All that is a very much larger undertaking, and probably a good deal less useful in general. The likes of X are a much more suitable abstraction.
The change here is implementing sendmsg and recvmsg: socket communications, implying communicating with something outside the VM, which in the context of GUI stuff almost certainly means an X server from outside.
With that and a X server using sixels, there would no longer be a need for something as complex as wslg for most apps: you could just export DISPLAY in your .bashrc, then use one Windows Terminal tab to start X apps, and the other one to display/interact with them.
WSL2 is already “blazingly fast”, and WSL was no slouch either. They are both impacted by storage overhead, which makes that tweet look kind of misguided.
I use Haskell for my ever continuing self-education and compiling and linking times with ghc under WSL2 is at least twice as slow as the same operations under native Linux on a much less capable machine.
If you cross the IO boundary to the Windows FS, yes, it is slow and your compilation will be slow. You need to have the project in the WSL FS, it's always been this way and I get pretty much native performance this way. Your editor can navigate to the WSL filesystem instead.
> If you cross the IO boundary to the Windows FS, yes, it is slow and your compilation will be slow
If you need good IO performance, WSL1 is still a valid option.
I need good IO performance, and it's not clear how much of the overhead is due to NTFS, and how much is due to 9p (to pass the VHDX file on the NTFS partition to WSL2)
Have you considered swapping Windows and Linux? Depending on your use case, you can still get pretty good performance out of virtualized Windows if you use PCI-E passthrough
I have: I generally prefer Windows, but depending on the test results, this option is definitely on the table!
As the computer has 2x NVMe and 2x GPU, it should be easy to do a GPU passthrough (with one GPU for Linux native, the other in passthough to Windows) but this brings a little extra complexity I'd rather avoid if I can.
WSL2 with code inside the Linux file system? Because if you're doing Linux work on the Windows file system, it will be bad. It is at native speeds inside the Linux file system though.
I remember no questions being asked when I set up WSL2.
I really doubt that WSL2 will be at native speed with the Linux file system, because reads and writes still have to go through Windows kernel to be mapped into actual hardware reads/writes.
> I really doubt that WSL2 will be at native speed with the Linux file system, because reads and writes still have to go through Windows kernel to be mapped into actual hardware reads/writes.
No, Linux is running alongside Windows, not under it. Windows itself is virtualized when WSL2 is turned on, and both Linux and Windows run under Hyper-V. It's quite fancy actually.
> Windows itself is virtualized when WSL2 is turned on
Is there documentation/explainer for this? I've been trying to understand more about this (was this a change in Windows 11? Is it only when WSL2 is installed?) for a while now, but not found anything apart from stray HN comments every now and then.
It's the fundamental difference between WSL1 and WSL2. As designed in the 90s, the Windows NT kernel can have different subsystems that offer different userspace APIs to different software. Originally there were the Win32 subsystem, the Security subsystem, the Posix subsystem and the OS/2 subsystem. The latter two fizzled out, but WLS1 is basically just another subsystem that provides a linux-compatible API to the Windows kernel. As it turns out providing full compatability is really hard though, so WSL2 is a complete rewrite, ignoring the subsystem concept. It basically just runs linux in Hyper-V, and adds some UI/UX to make that convinient.
That Windows is then running virtualized is just how Hyper-V works, as soon as you turn on Hyper-V the host Windows runs as a guest in Hyper-V, though with special privileges [1]
> WLS1 is basically just another subsystem that provides a linux-compatible API to the Windows kernel
WSL1 is not an NT subsystem, either. WSU (the POSIX subsystem) was an actual NT subsystem, but with WSL, they abandoned this concept and went with a different design for performance reasons.
> I remember no questions being asked when I set up WSL2.
If the (WSL) path to your project is inside ~/ it's on the linux file system. If it's in /mnt/c/ it's in the Windows filesystem. In WSL1 those were about equally fast, both going throught the windows kernel. With WSL2, accessing /mnt/c/ from Linux has to go through Windows, and accessing \\wsl$ from Windows has to go through Linux, so where you put the files decides where they are fast.
> In WSL1 those were about equally fast, both going throught the windows kernel
Yes, WSL1 was great. I hope it will never be sunset, as I need good IO performance.
> where you put the files decides where they are fast.
I also also hope we'll eventually get an option to pass partitions to HyperV, which is currently possible only with full disks.
It would be a return to the previous WSL1 level of performance for files within a NTFS partition: even if you'd need a separate NTFS partition, that'd be a minor nuisance to guarantee performance.
> I really doubt that WSL2 will be at native speed with the Linux file system
I had the same doubts, but when I finally took the time to install it, I found
out that I was wrong. One Java program that I use, which is very heavy
on the filesystem and slow like hell under Windows it a LOT faster under WSL2.
As Java profiler support is poor on Windows, I now use WSL2 to run async-profiler
sessions and get results as accurate as on native Linux.
EDIT to add: you really have to run things fully in the WSL2 filesystem, be it for compiling
or using a program. The performance is awful if you run WSL2 processes under the Windows
filesystem.
I agree. If I would stay solely inside the WSL2 VM, it might be fast but my problem is that I need a tight integration between the Windows and the WSL2 file systems. In effect, this is a real pain, especially when it comes to I/O. But WSL2 also constantly loads one or more CPU cores to 100%+. My whole system feels handicapped. I probably have additional problems due to MS Defender, all known measures (that I can implement at all in a corporate-managed environment) have hardly remedied it.
VMWare is a must for Windows. Code compiles 10x faster in a virtual machine on Windows than it does if you compile on Windows natively. Or 100x faster if you forgot to disable Windows Defender. I've always found that amusing.
I use VMWare Workstation Player to run a bunch of different operating systems on Windows, e.g. FreeBSD, OpenBSD, Windows 7, etc. It's served my needs pretty well for some time. Hyper-V as I understand it is pretty new. I think I'm using it to run WSL2 and I'm pretty happy with its performance. However I don't know how to use Hyper-V as a VMWare replacement. I'd imagine if it's able to do that, the learning curve would be a bit higher due to the CLI.
Most compilation processes create a lot of very small files. Or a metric ton of them if you're unlucky enough to do js.
NTFS is awful at doing lots of small accesses on very small files, at least compared to Ext4. Add to that the fact that if you're on WSL, it needs to write POSIX attributes (which NTFS has no support for and therefore WSL must emulate), and that's a recipe for builds being slowed massively by IO.
As for Defender, well, Defender sees new files and halts everything while it scans them. You can see how that can be a problem for a process that pops out tens of thousands of temporary files, which it will all go and scan, locking them while it does so.
Another thing that makes Windows builds go slow is the lack of vfork(). On Linux, vfork() lets you create and teardown a new process in ~30 microseconds. On Windows it takes about 6000 microseconds to do the same thing, which is in large part due to the lack of copy-on-write memory (which Windows won't support, due to Microsoft corporate culture being heavily biased against overcommit models).
I wonder if blink could be used as a lightweight sandbox. Looking at PR46[0], it seems sandboxing is not one of the current features, but it would be cool to have a way to run arbitrary code (e.g: Python) in a sandboxed environment. Even cooler if you could limit the amount of memory/CPU used.
Author of Blink here. I wrote another command called pledge.com which does sandboxing: https://justine.lol/pledge/ Let's say for example you want to be able to run the `ls` command under Blink in a way that (1) restricts which parts of the file system it's allowed to use and (2) prevents it from talking to the Internet. You could say:
You can now be certain that your `ls` command isn't spying on you or uploading your bitcoin wallet to the cloud. The blink command itself is currently unsecured. However that shouldn't matter, since we can compose blink with pledge.com to bolt on all the security we need separately!
Awesome! I was just reading about pledge. I think I need to find some time to play around with it. Right now, there's a lot of interest in using WASM as a sandbox for untrusted code, but pledge looks easier to use and more versatile (you don't need any kind of WASM compatibility, just a binary to run).
Offline WASM runtimes like Wasmtime are pretty cool, however I'd call SECCOMP BPF with Landlock LSM a shining beacon of light. The problem is that (1) coding BPF assembly has always been terrifying to the uninitiated and (2) Landlock LSM only came out in the past year. I believe pledge.com is the first tool that makes using both these Linux security tools together universally accessible for everyone. Be sure to run a bleeding edge Linux kernel if you use the `-v PATH` flag. If you're running an older kernel, then the pledge.com command will treat unveiling as a no-op in the interest of compatibility. I use Landlock LSM on my desktop, which runs Alpine Linux 5.15.74-0-lts. I also use Landlock in production on GCE, but I needed to `apt install linux-image-5.18.0-0.deb11.4-cloud-amd64` in order for it to work.
Moving to Wayland does not prevent X11 apps. And I don't think anyone would expect this to be a good solution for crossplatform guis. More of a fun starting point.
It depends what you mean. For example, I can run `blink curl ...` on Alpine Linux and blink will run the the dynamically linked curl binary that Alpine provides in /usr/bin/curl. So blink programs can leverage system libraries on x86_64 systems. What Blink does not do is translate dynamic ABIs. Here's what I mean by that. Let's say I write an x86_64-linux binary that calls dlopen(). It'll work on Linux. But if I scp that Linux program over to an Apple system, then it's not going to be able to dynamically open MachO .dylib files, because Blink only translates man(2) SYSCALL APIs. It would be able to dynamically load any Linux .so files I copied over though. Some of our users have experimented with this in cool ways. One contributor created a chroot'd folder in a Cygwin environment that contains an rsync'd Alpine Linux userspace (binaries, libraries, configs, and all) and then used blink to launch its /bin/bash shell. This astoundingly produced a mostly working Linux system inside Cygwin!
PowerPC is one of the 11 hardware architectures on which we regularly test Blink using Qemu https://github.com/jart/blink/blob/c8dbbc8cf553680ab48d8a304... so my guess would be yes. However it depends on how old the Mac software is. Blink currently assumes modern POSIX system calls are available, e.g. fstatat(), O_CLOEXEC, etc. i.e. you currently need a host operating system published after ~2010ish. That could change in the future if we end up writing a ./configure script.
"In March 2014, Tunney petitioned the US government on We the People to hold a referendum asking for support to retire all government employees with full pensions, transfer administrative authority to the technology industry, and appoint the executive chairman of Google Eric Schmidt as CEO of America"
Justine also wrote Rampage toolz 2.0, one of the early "AOL Progz" [1]. Can't quite remember which ones I used as a kid back then.
She also claimed to have written hacks for Kevin Mitnick.
Justine was somewhat controversial inside the Occupy movement, she was outside most circles and was criticized for extremely strongly-voiced technocratic opinions (see "Nerds Should be Segregated" [2], since deleted), but she's incredibly, incredibly talented. A good profile of Justine x Occupy post-mortem is here [3]
Author of Blink here. Yes. Absolutely. I've always wanted to have Java's tools (e.g. "build once run anywhere" simplistic development) except I want to program in C / C++ instead of Java. The main platform I target is x86_64 desktops and servers, on which I want to run natively (rather than Java, which would force my code to always be virtualized) and Blink lets me pull out the virtual machine trick when I need it (e.g. someone wants to run the apps I've released on an esoteric platform like MIPS OpenBSD). All the x86_64 Linux executables I spent the past several years writing and releasing are suddenly able to be enjoyed by users on platforms I'd never dreamed of, like Haiku, and I didn't have to do anything to "port" and re-release those binaries on my website in order to do that. We just had to build one tiny 180kb virtual machine (unlike the JVM which is more like 100mb and might bundle a browser toolbar in the installer) and I made it open source (ISC, unlike Qemu which is GPL) so that you can enjoy these benefits too.
Somebody should write fanfic where jart goes back in time and forestalls the Java revolution in ~roughly 1995 by releasing Blink. Instead of the entire several decades of Java (and Javascript), Blink becomes the technology underlying web (the browser is really just a portable presentation layer) server and client programming. Lightweight phones with long battery life and portable architectures revolutionize mobile computing 5 years earlier. World GDP is 35% higher, and life is generally happier, and much easier.
(seriously though, thanks for doing all this. I like your philosophy compared to what the world ended up with as the dominant paradigm)
I still find it surreal that these operating systems share so many mechanisms that these portable EXEs are even possible.
Someone should write some kind of historical explanation of how this could have happened. And how critical is it for the "APE" scheme to work? Could APE binaries work on AROS x86_64 for instance? (Without emulation.)
I also find it surreal that it took this long for someone to do something like Actually Portable Executable. I mean, it's slick and easy to use, but I haven't even seem assembler demos of this.
Is the same polyglot EXE possible also on 32-bit x86? If so, I find it even weirder this have not been done long ago, considering the strange hacks people have pulled over the years.
I'm fascinated by the whole thing, as you can tell. Back in the day, Windows and Unix seemed so far from each other, like oil and water.
In this blog post https://justine.lol/sizetricks/#elf I talk about how simple it is to create a single file binary from scratch that runs on Linux, FreeBSD, OpenBSD, and NetBSD. All Actually Portable Executable does is it takes that trick three steps further, to support Windows, MacOS, and BIOS from a single file binary too. That requires a bit more effort.
These are simply the operating systems I knew best, because I grew up using them, and as a result was able to discover the right combination of tricks to get it to work. I'm sure someone else who loves AROS and knows AROS would also be able to discover a way to trick it out so that APE could support it too. Same goes for other platforms. For example, one of the things on my TODO list this year is to find out if the APE x86_64 hacks are replicable for the ARM64 architecture.
The closest thing to a scientific theory behind what APE is would be what the information security community calls a "polyglot". Researching that is another great way to learn more similarly inspired tricks.
> Is the same polyglot EXE possible also on 32-bit x86? If so, I find it even weirder this have not been done long ago, considering the strange hacks people have pulled over the years.
It's actually even easier to do for i386 than it is for x86_64. The blog post above explains why that's the case, due to the syscall magic numbers being more similar. Also I'm equally astonished no one did this before I came along. I think part of why that's the case, is that I couldn't just write APE. Doing this project meant that I had to implement a C library from scratch too, which is a years long coding marathon.
I think for something like AROS the combination of support for ELF and such a small community would mean that if someone wanted to do this for AROS the easier path might well be to add support directly in the loader.
Thanks, great explanations. I think the most astonishing thing to me is that I never saw such a polyglot EXE even print "hello" on stdout, written in assembler.
> Blink lets me pull out the virtual machine trick when I need it (e.g. someone wants to run the apps I've released on an esoteric platform like MIPS OpenBSD).
Me and the other user of Mips on OpenBSD are both glad you do!
curious about "except I want to program in C / C++ instead of Java", does that mean that you prefer writing C code instead of e.g. Rust or Zig?
In other words, what is your sincere opinion about Rust or Zig?
Thank you!
Theoretically, if Sun and Oracle would have cared for the experimental runtimes which compiled to native instead of JVM. I think GNU had one going, but RMS hated that too, so there was kind of a gap in the marketplace for such a thing. You can sort of do it with .NET now.
Edit: I completely misunderstood what Blink is in relation to WASM. I'm leaving my original comment below, but it is quite wrong.
Still, Blink is a VM in the same sense as the JVM, and it does JIT compilation (or it can interpret program ASM) just as much as that. As far as I understand from the git page, it is using x86_64 assembly as the "bytecode" though, so the JIT step may be quite trivial if running on the same arch and/or OS.
Old post:
This isn't compiling to native either, not ahead of time anyway. It is compiling C or C++ to WASM, and then running that in a browser. The browser will JIT compile the WASM, just like the JVM would.
The biggest difference is that everyone has a browser, while not everyone has a JVM pre-installed.
This doesn't use WASM or a browser at all. This is an new, original x86-64 machine code interpreter with a JIT compiler that supports x86-64 and aarch64. This is an extension of Justine's APE project to use x86-64 ELF as the core of write-once-run-anywhere software.
We did, but not in this project. :-) This is indeed more like an embedded JVM, not totally unlike early J2ME, but with x86_64 instead of JVM instructions.
not at all a critique on your project, if that was the impression I gave :)
I'm old enough to remember when Java applets were invented, grew up using Visualage Smalltalk on OS/2 at school, and am still bitter that nobody ever said: but this is Smalltalk, why don't we use Smalltalk then? :)
Because Smalltalk big supporters like IBM, decided to repurpose their Smalltalk offerings into Java.
Visual Age Smalltalk had a role similar to .NET on the OS/2 ecosystem, SOM (OS/2 version of COM) supported Smalltalk, implementation inheritance and meta-classes, so yeah it didn't help to go full on Java.
Also another Visual Age thing, Visual Age for C++ v4 also had a Smalltalk like experience for C++, unfortunely it was too demanding in resources.
Smalltalk was ahead of its time. So much so that its VM was unbearably slow. Thirsting for CPU power that just wasn't there. I recall using a Pentium-based machine and waiting up to five seconds to open a context menu, and the IDE (VisualWorks?) could not keep pace with my typing. I could type in a message name and watch the letters appear one-at-a-time. Compared with Borland's TurboPascal and similar IDEs of the day, Smalltalk was a slog, albeit a powerful one.
There are other issues with Smalltalk that we know now that we didn't then. Such as broad and deep class hierarchies are hard to use. Smalltalk didn't enforce type-safety when I used it (i.e., you can have runtime faults because you can send any message to any class and if you didn't implement "does not understand", you won't know until exercising that branch).
Trademark protection is around market concerns, and in general, open source projects side-step questions of "market" by giving everything away for free.
... apart from that, it's a legit hard problem to name things. As in "One of the two recognized hardest problems in software engineering" (alongside cache management and off-by-one errors).
Maybe you build proton and have any windows game (or software?) in the browser?