Pretty sure she was the first place finisher overall. I am not sure how much of a gender advantage there is over those distances. Similarly, age is less of a factor too, with older athletes regularly beating those who are a decade younger.
Specifically, the blog post is about debugging the test script like a normal Python program: the standard method to run the test interactively is somewhat clunkier.
Anyway, It's a very niche and technical article about NixOS, I wonder why it was at the top of HN for hours with only 3 bot comments.
> Is this only about Python applications or general?
It has nothing to do with Python, it just happens to use it (previously it was written in Perl). The NixOS test framework is a way to test NixOS configurations: it uses a Qemu VM to run NixOS and a Python script to perform actions (check that a file exists, run a command that should succeed, take a screenshot, etc.).
It's basically an integration test suite for an OS. It's pretty great, if you ask me: you control the entire machine from the boot loader to the desktop environment. You can even network several machines together and test client/server or p2p scenarios.
> How does this compare to testing / debugging flows on Debian, Alpine, CentOS?
I have no idea, but I would be surprised if they have something this sophisticated. Here's some cool ones:
When you're adding some module code to NixOS, you should write a test with this.
More generally, if you're building some complex system with many interacting parts and you want to test it extensively, this is a good way.
> Conceptually, during a door invocation the client thread that issues the door procedure call migrates to the server process associated with the door, and starts executing the procedure while in the address space of the server. When the service procedure is finished, a door return operation is performed and the thread migrates back to the client's address space with the results, if any, from the procedure call.
Note that Server/Client refer to threads on the same machine.
While I can see performance benefits of this approach, over traditional IPC (sockets, shared memory), this "opens the door" for potentially worse concurrency headaches you have with threads you spawn and control yourself.
Has anyone here hands-on experience with these and can comment on how well this worked in practice?
IIUC, what they mean by "migrate" is the client thread is paused and the server thread given the remainder of the time slice, similar to how pipe(2) originally worked in Unix and even, I think, early Linux. It's the flow of control that "conceptually" shifts synchronously. This can provide surprising performance benefits in alot of RPC scenarios, though less now as TLB, etc, flushing as part of a context switch has become more costly. There are no VM shenanigans except for some page mapping optimizations for passing large chunks of data, which apparently wasn't even implemented in the original Solaris implementation.
The kernel can spin up a thread on the server side, but this works just like common thread pool libraries, and I'm not sure the kernel has any special role here except to optimize context switching when there's no spare thread to service an incoming request and a new thread needs to be created. With a purely userspace implementation there may be some context switch bouncing unless an optimized primitive (e.g. some special futex mode, perhaps?) is available.
Other than maybe the file namespace attaching API (not sure of the exact semantics), and presuming I understand properly, I believe Doors, both functionally and the literal API, could be implemented entirely in userspace using Unix domain sockets, SCM_RIGHTS, and mmap. It just wouldn't have the context switching optimization without new kernel work. (See the switchto proposal for Linux from Google, though that was for threads in the same process.)
There isn't a door_recv(2) systemcall or equivalent.
Doors truly don't transfer messages, they transfer the thread itself. As in the thread that made a door call is now just directly executing in the address space of the callee.
> Doors truly don't transfer messages, they transfer the thread itself. As in the thread that made a door call is now just directly executing in the address space of the callee.
In somewhat anachronistic verbiage (at least in a modern software context) this may be true, but today this statement makes it sounds like code from the caller process is executing in the address space of the callee process, such that miraculously the caller code now can directly reference data in the callee. AFAICT that just isn't the case, and wouldn't even make sense--i.e. how would it know the addresses without a ton of complex reflection that's completely absent from example code? (Caller and callee don't need to have been forked from each other.) And according to the Linux implementation, the "argument" (a flat, contiguous block of data) passed from caller to callee is literally copied, either directly or by mapping in the pages. The caller even needs to provide a return buffer for the callee's returned data to be copied into (unless it's too large, then it's mapped in and the return argument vector updated to point to the newly mmap'd pages). File descriptors can also be passed, and of course that requires kernel involvement.
AFAICT, the trick here pertains to scheduling alone, both wrt to the hardware and software systems. I.e. a lighter weight interface for the hardware task gating mechanism, like you say, reliant on the synchronous semantics of this design to skip involving the system scheduler. But all the other process attributes, including address space, are switched out, perhaps in an optimized matter as mentioned elsethread but still preserving typical process isolation semantics.
If I'm wrong, please correct me with pointers to more detailed technical documentation (Or code--is this still in Illuminos?) because I'd love to dig more into it.
I didn't imply that the code remains and it's only data that is swapped out. The thread jumps to another complete address space.
It's like a system call instruction that instead of jumping into the kernel, jumps into another user process. There's a complete swap out of code and data in most cases.
Just like with system calls how the kernel doesn't need a thread pool to respond to user requests applies here. The calling thread is just directly executing in the callee address space after the door_call(2).
> Did you mean door_call or door_return instead of door_recv?
I did not. I said there is no door_recv(2) systemcall. The 'server' doesn't wait for messages at all.
I think what doors do is rendezvous synchronization: the caller is atomically blocked as the callee is unblocked (and vice versa on return). I don't think there is an efficient way to do that with just plain POSIX primitives or even with Linux specific syscalls (Binder and io_uring possibly might).
The thread in this context refers to kernel scheduler thread[1], essentially the entity used to schedule user processes. By migrating the thread, the calling process is "suspended", it's associated kernel thread (and thus scheduled time quanta, run queue position, etc.) saves the state into Door "shuttle", picks up the server process, continues execution of the server procedure, and when the server process returns from the handler, the kernel thread picks up the Door "shuttle", restores the right client process state from it, and lets it continue - with the result of the IPC call.
This means that when you do a Door IPC call, the service routine is called immediately, not at some indefinite point in time in the future when the server process gets picked by scheduler to run and finds out an event waiting for it on select/poll kind of call. If the service handler returns fast enough, it might return even before client process' scheduler timeslice ends.
The rapid changing of TLB etc. are mitigated by hardware features in CPU that permit faster switches, something that Sun had already experience with at the time from the Spring Operating System project - from which the Doors IPC in fact came to be. Spring IPC calls were often faster than normal x86 syscalls at the time (timings just on the round trip: 20us on 486DX2 for typical syscall, 11us for sparcstation Spring IPC, >100us for Mach syscall/IPC)
EDIT:
[1] Some might remember references to 1:1 and M:N threading in the past, especially in discussions about threading support in various unices, etc.
The "1:1" originally referred to relationship between "kernel" thread and userspace thread, where kernel thread didn't mean "posix like thread in kernel" and more "the scheduler entity/concept", whether it was called process, thread, or "lightweight process"
Sounds like Android's binder was heavily inspired by this. Works "well" in practice in that I can't recall ever having concurrency problems, but I would not bother trying to benchmark the efficiency of Android's mess of abstraction layers piled over `/dev/binder`. It's hard to tell how much of the overhead is required to use this IPC style safely, and how much of the overhead is just Android being Android.
Not sure which one came first, but Binder is direct descendant (down to sometimes still matching symbol names and calls) of BeOS IPC system. All the low level components (Binder, Looper, serialization model even) come from there.
From what I understand, Sun made their Doors concept public in 1993 and shipped a SpringOS beta with it in 1994, before BeOS released, but it's hard to tell if Sun inspired BeOS, or of this was a natural solution to a common problem that both teams ran into at the same time.
I'd expect convergent evolution - both BeOS team and Spring team were very well aware of issues with Mach (which nearly single-handedly coined the idea that microkernels are slow and bad) and worked to design better IPC mechanisms.
Sharing of scheduler slice is an even older idea, AFAIK, and technically something already done whenever you call a kernel (it's not a context switch to a separate process, it's a switch to different address space but running in the same scheduler thread)
Binders has been in mainline kernel for years, and some projects ended up using it, if only to emulate android environment - both anbox and its AFAIK successor Waydroid use native kernel binder to operate.
You can of course build your own use (depending on what exactly you want to do, you might end up writing your own userland instead of using androids)
As far as I understand, it is already mainlined, it's just not built by "desktop" distributions since nobody really cares - all the cool kids want dbusFactorySingletonFactoryPatternSingletons to undo 20 years of hardware performance increases instead.
Be Book, Haiku source code, and yes Android low level internals docs.
A quick look through BeOS and Android Binder-related APIs will quickly show how Android side is derived from it (through OpenBinder, which was for a time going to be used in next Palm system based on Linux, at least one of them)
Think of it in terms of REST. A door is an endpoint/path provided by a service. The client can make a request to it (call it). The server can/will respond.
The "endpoint" is set up via door_create(); the client connects by opening it (or receiving the open fd in other ways), and make the request by door_call(). The service sends its response by door_return().
Except that the "handover" between client and service is inline and synchronous, "nothing ever sleeps" in the process. The service needn't listen for and accept connections. The operating system "transfers" execution directly - context switches to the service, runs the door function, context switches to the client on return. The "normal" scheduling (where the server/client sleeps, becomes runnable from pending I/O and is eventually selected by the scheduler) is bypassed here and latency is lower.
Purely functionality-wise, there's nothing you can do with doors that you couldn't do with a (private) protocol across pipes, sockets, HTTP connections. You "simply" use a faster/lower-latency mechanism.
(I actually like the "task gate" comparison another poster made, though doors do not require a hardware-assisted context switch)
Well, Doors' speed was derived from hardware-assisted context switching, at least on SPARC. Combination of ASIDs (which allowed task switching with reduced TLB flushing) and WIM register (which marked which register windows are valid for access by userspace) meant that IPC speed could be greatly increased - in fact that was basis for "fast path" IPC in Spring OS from which Doors were ported into Solaris.
I was (more) of a Solaris/x86 kernel guy on that particular level and know the x86 kernel did not use task gates for doors (or any context switching other than the double fault handler). Linux did taskswitch via task gates on x86 till 2.0, IIRC. But then, hw assist or no, x86 task gates "aren't that fast".
The SPARC context switch code, to me, always was very complex. The hardware had so much "sharing" (the register window set could split to multiple owners, so would the TSB/TLB, and the "MMU" was elaborate software in sparcv9 amyway). SPARC's achilles heel always were the "spills" - needless register window (and other cpu state) to/from memory. I'm kinda still curious from a "historical" point of view - thanks!
The historical point was that for Spring OS "fast path" calls, if you kept register stack small enough, you could avoid spilling at all.
Switching from task A to task B to service a "fast path" call AFAIK (have no access to code) involved using WIM register to set windows used by task A to be invalid (so their use would trigger a trap), and changing the ASID value - so if task B was already in TLB you'd avoid flushes, or reduce them only to flushing when running out of TLB slots.
The "golden" target for fast-path calls was calls that would require as little stack as possible, and for common services they might be even kept hot so they would be already in TLB.
So if I understand it correctly, the IPC advantage is that they preserve registers across the process context switch, thereby avoiding having to do notoriously expensive register saves and restores? In effect, leaking register contents across the context switch becomes a feature instead of a massive security risk. Brilliant!
Why would you care who spawned the thread? If your code is thread-safe, it shouldn't make a difference.
One potential problem with regular IPC I see is that it's nondeterministic in terms of performance/throughput because you can't be sure when the scheduler will decide to run the other side of whatever IPC mechanism you're using. With these "doors", you bypass scheduling altogether, you call straight "into" the server process thread. This may make a big difference for systems under load.
Good to see LU decomposition with full pivoting being implemented here (which is missing from BLAS/LAPACK). This gives a fast, numerically stable way to compute the rank of a matrix (with a basis of the kernel and image spaces). Details: https://www.heinrichhartmann.com/posts/2021-03-08-rank-decom....
If you are going to include vs MKL benchmarks in your repo, full pivoting LU might be one to consider. I think most people are happy with partial pivoting, so I sorta suspect Intel hasn’t heavily tuned their implementation, might be room to beat up on the king of the hill, haha.
it might be interesting to add butterfly lu https://arxiv.org/abs/1901.11371. it's a way of doing a numerically stable lu like factorization without any pivoting, which allows it to parallelize better.
Is there a collection of algorithms published somewhere, in a coherent manner? It seems like to me, there should be a collection of matrix algebra, symbolic algebra, etc. algorithms somewhere that makes it easy to implement these in any language or system. As it is, it seems you need to already be an expert or researcher in the space to make any movement.
>The BLAS-like Library Instantiation Software (BLIS) framework is a new infrastructure for rapidly instantiating Basic Linear Algebra Subprograms (BLAS) functionality. Its fundamental innovation is that virtually all computation within level-2 (matrix-vector) and level-3 (matrix-matrix) BLAS operations can be expressed and optimized in terms of very simple kernels.
That can be argued, but they're pretty esoteric. For example, what language are they written in? Fortran, assembly, etc.? Probably nothing the majority of developers or mathematicians can't understand. And creating bindings for them is non-trivial. I don't even know where to start from the website. And on the website, all the links to the routines are broken. https://www.netlib.org/blas/
The naming convention is a bit esoteric, but once you figure it out it's not too bad. I don't think writing bindings is too hard once you know what's going on. You most likely would want to write some kind of wrapper on top of that in order to make things a little more user-friend.
I will agree with you on the routines being broken. That happened within the past year. Not sure why it happened, but it's annoying. If you know the name of the routine you can usually find documentation on another site.
74xx series ICs (eg. 74ACT family in case of the cpu, if I read correctly).
More generally, it may refer to "basic logic elements whose function is easily inspected".
> Why would this prevent interrupts?
Not at all - in theory.
In practice, interrupt support tends to complicate cpu designs. Complicate = more logic = more ICs. So builder decided against it & chose not to implement interrupts on the cpu.
Is this based on Hemlock, or was it written from scratch?
I really, really want a graphical emacs. By that I mean, I want an emacs with graphics, not an emacs in windows. I want to easily be able to draw stuff.
I ran into this the other day, I like to dabble in emacs, and I had a list of results I wanted to make a quick chart. I ended up dumping it out in a simple CSV, and copy/pasting that into a spreadsheet, and hitting "chart".
Would have been nice to go (line-chart my-list) and have a window pop up with reasonable defaults. And be able to print it to a PDF.
emacs has some SVG support, but I guess its maintainer whim whether you get it or not, I wasn't able to easily get it working on my Mac. That could be a start. But something "nicer" would be, you know, "nice".
When building it, choose “make sdl2” to get the graphical UI. I also build the text UI, but the sdl2 UI is more fun and is very responsive. I didn’t see any ability to draw graphs in the window.
I can understand wanting to restrict use of something a person created using a restrictive licence. I can’t understand being sad about someone else not restricting what they created. To me it looks like someone being sad after seeing someone else giving out free ice cream, even though the person being sad didn’t contribute to making the ice cream.
My view on the GPL is that it's freedom-preserving, like how a law forbidding the enslavement of people is freedom-preserving, even though it technically restricts a freedom. That's why I look more favourably on GPL software than software with a so-called 'permissive' license.
This and similar arguments in this thread work on the same logic that frames software piracy as theft. It’s the fallacy of treating software as a tangible thing that, when copied, is taken away from someone and altered irreversibly.
That's ridiculous. The GPL explicitly allows people to make copies.
What many free software advocates actually want is a world without copyright on software. It's copyright that frames copying as stealing. In a world with copyright, the only way to counter that is copyleft. The fact GPL can also enforce inclusion of source code is a nice side effect of copyleft. But ultimately what we want is a world where software is shared, treated as knowledge and not as a product.
MIT style licences are just not good enough. That software could still end up in some copyright protected proprietary product one day and then, yes, someone could accuse me of "stealing" their property, which contains our software.
> What many free software advocates actually want is a world without copyright on software.
Some of them sure. The subset of them who are GPL advocates are simply confused though. GPL is not possible in a world without copyright. In a world without copyright I can release software, which can be written from the ground up or be based on something open source, doesn’t matter, and not share the code with anyone. Just the binary. I can also create AWS based on open source software and nothing like AGPL will stop me from providing proprietary services based on it.
And the reason I pointed to the fallacy is that multiple people in this thread suggest that making a proprietary product based on open source software constitutes taking something away from users of this software. This is illogical. They still have the original. The only person who loses in this situation is the original author, not the users.
You and the author appear to be the same guy. To be sure, the piece itself is an academic exercise hypothesizing a world in which the MIT license were law. And in such a world, the GPL gestapo would continue railing against opaque binaries even if those binaries were shameless but legal ripoffs of someone's hard work.
If I took emacs, modified it to be thread-safe, and distributed it as an opaque binary, John Q. Emacs User retains his thread-unsafe version but now knows there's something better out there which he can't iterate on. You can say this knowledge isn't a real "loss," but that would be dismissing the concept of opportunity cost.
So you’re fine with software provided as opaque binaries without source code or only as SaaS? Because without copyright you’re going to get more of that and no licence is going to stop that.
No, of course I'm not ok with that! But trying to get rid of copyright for software and make access to source code a right is a huge task. That's why the GPL is a pragmatic solution. The GPL is something we can use today to build a foundation of free software and make the idea of software freedom normal and expected. This smooths the way towards a future where we don't need the GPL, but we've got a long way to go yet.
Licences like MIT do absolutely nothing up further this goal. They don't even try to do it today even though we know it can be done. It seems convenient for software you might use now, but to contribute to it? It doesn't give developers any guarantee that their work will be received as free software by future users. You're essentially working for corporations for free. I want my work to be available for everyone.
And even the threat of that (and *GPL’s viral nature) makes it INCREDIBLY unlikely legal teams allow employees to even use binaries of the code on their workstations. Of course nothing is actually stopping them, and it would be hard to prove it if they did, but there is at least some semblance of a threat.
MIT and similar are simply roundabout ways to make it public domain.
I care a great deal for the GPL, but I'd be fine with copyright if it wasn't so stupidly long. The reason I care about the GPL is my belief in an informed public. As far as I'm concerned, withholding vital information (like a list of ingredients or source code) from the public should be a criminal offence.
Whilst yes, you could technically say 'well, if the user wants to know what this software is doing, they can just disassemble it', I'd consider that malicious compliance. It's like listing ingredients by their scientific names (like 'Arachis Hypogaea' instead of 'peanut').
For those confused by the dangling modifier, he means the combined act of cutting the trees and selling the wood is restrictive. Also, a terrible analogy since trees are an exhaustible resource within a meaningful time range.
> ...even though the person being sad didn’t contribute to making the ice cream.
But the issue of non-GPL software is the person on the receiving end might want to contribute, be technically able to contribute and indeed need to contribute to be able to keep using the software ... but then be blocked for legal reasons. The GPL doesn't care if you sell software or give it away or whatever, it is all about protecting freedom over time after the initial installation has happened.
The thing that makes the ice cream analogy ok is that the ice cream is literally synthesised into your body and comes under your complete control. If the ice-cream seller retained any sort of control over the ice cream after you'd ingested it that would be a serious problem! Both technically and morally. The GPL is aligning what happens in software to the analogy.
The poster I replied to complained about someone choosing MIT over GPL, not a proprietary licence over GPL. Noone is blocked from contributing for legal reasons. Quite the opposite.
The point of the MIT license is to make it easier to link it to proprietary software. It is setting up a situation where users will not be free. In and of itself it isn't a bad license; it just doesn't protect freedom and it can be used in ways that block people from contributing. By design.
If you expect people to treat MIT licensed software as GPL licensed software, then they should be licensing it under the GPL. That they aren't implies that the intent is different.
That they aren't implies that the intent is different.
News flash buddy. OSS devs, college and graduate students mostly, are clueless about the distinction and select whatever license tickles their fancy from GitHub's dropdown menu. They see a lot of MIT licenses floating around, recognize "MIT" as a good university, and figure they can't go wrong with it. They, like most normies, don't see how "freedom" figures into software at all. All they care about is getting a little shine before they join a FAANG after graduation.
As for why someone who does understand the distinction would choose MIT over GPL, ask Linus about why he'd "never want anything to do with the FSF" after being told to switch to GPLv3. Yes, MIT versus GPL is not the same as GPLv2 versus GPLv3 but the general concepts are the same.
If you want to argue that licenses are picked randomly then OK. But in that case, people should pick the GPL; it is a better license for preserving the freedom of software users than the MIT license.
> ... ask Linus about...
Linus uses GPLv2. I personally see merit in the argument that v2 is better than v3. Neither is MIT.
If you want to use the GPL license and not associate with the FSF then cool. If Linus wants to do that, also cool. The issue is the license, not if you want to work with the FSF. The GPL also grants users freedom from the FSF.
> It is setting up a situation where users will not be free.
No. It is setting up a situation where users can choose between a free option A and a proprietary option B based on A. Being sad about A being too liberal is suggesting it’s more important for you that B doesn’t exist than that A exists.
> If you expect people to treat MIT licensed software as GPL licensed software,
I don’t. Contrary to GNU and FSF websites, GPL is not a synonym for freedom. Most people, outside of GNU-centric places like this post about Emacs, prefer more liberal licences.
But it sounds like we have found agreement on the point that the MIT license is setting up an option B, where a person is blocked from contributing for legal reasons?
That is the big difference between MIT and GPL. With MIT licensed software, sometimes you have software that you can't legally adjust (or help others adjust). With GPL ... I mean it might be technically possible with some wildly creative approach but I haven't heard of such a thing.
> Most people, outside of GNU-centric places like this post about Emacs, prefer more liberal licences.
Controlling someone else's computer through legal means isn't liberal. I'm sure a liberal could call for that and accept a little ideological impurity.
If software is property, it has a new owner after it is sold and they can do what they like with it. That isn't what copyright does, it creates some sort of Frankenstein permanent-rent concept where the owner often doesn't maintain anything that is at odds with technical and market realities, relying heavily on the prosecution of victimless crimes. Which although arguably a desirable thing (I don't think so myself though) is illiberal.
The MIT license is explicitly designed to be compatible with building proprietary software. The key point of difference from the GPL is that you can use MIT code to create proprietary software.
I'm no lawyer, but it isn't even obvious to me that you have to MIT license a binary created directly from MIT licensed source code. "Software" seems to be the source code and therefore the binary probably isn't a copy or substantial portion of the MIT licensed program. Unless lawyers are redefining words on me it looks like I can compile MIT licensed source and distribute it under whatever alternate license I like.
But regardless, if the point of the license is to enable creating proprietary software, it is no stretch at all to speculate that it'll be used to create proprietary software.
I can't understand being sad about someone not restricting what they created
I can. The guy giving away the ice cream is destroying the market. You can't compete against free. If programmers would stop surrendering their labor, personal data resellers like Google and Facebook wouldn't be our primary means of making a comfortable living. Say what you will about Microsoft's monopoly in the 90s. At least they took your money with your eyes open.
The irony of course is free software zealots are pissed at the free ice cream guy for completely different, and I agree with you, utterly stupid reasons.
Yeah. I can get this angle. This is why we don’t have small independent companies making C and C++ compilers today. Only BigCorps which fund this development with money made from less honourable business.
I call these personality types free software socialists, and free software communists. The free software socialists are my friends. The free software communists are not.
I suspect that if someone is in a position where they are compelled to purchase and use the proprietary version and not the free (and available) version, they already lack most software freedoms. Probably several other, more important, freedoms as well.
Although I agree, notice that the only act the GPL compels someone to do is to hand over the source code with the program.
MIT doesn't give people a freedom to restrict because that isn't a freedom. The restriction is always done by the legal system. The GPL is just an attempt to stop the legal system from interfering in the market to restrict user freedom. So both licenses are actually communicating to a 3rd party (a judge) under what circumstances they should restrict the freedom of others. GPL says to stay out of it as long as people are sharing their source code and MIT says get involved sometimes/its complicated.
It isn't a practical difference, but it is philosophically important. The amount of personal freedom both licenses give is technically quite similar.
MIT doesn’t give anyone the ability to restrict the freedom of others. When you fork a project and use a difference licence for it, the original project remains under MIT.
I started my blog in 2012, when I shifted my academic career from Mathematics to Computer Science. This topic was literally the first thing that I studied:
Never regretted going down this deep rabbit hole. IIRC, Julia also has a math background. Maybe it's the desire for bottom-up reasoning that leads math folks
towards experiments like this. Great to see her making this approachable for a large audience.
I noticed you mentioned leaving the leadership team in addition to the Board of Directors in 2021 (which was before Hashicorp changed their projects' license to Business Source License). Were you one of the champions for OSS at Hashicorp during your time in leadership there?
And can you speak to whether the leadership's new direction was a factor in your decision to leave the company?
For Courtney Dauwalter are we talking "won" as in across all starters or across all women starters?
Winning against a mixed field would be beyond impressive!