An approach for this kind of problem: suppose you have a very simple language composed of simple, verifiable instructions, like: block read, block write, read-offset, write-offset, simple arithmetic, simple branching; you could compose programs for various file-system operations and pass them over to the kernel for execution en masse, rather than needing a chatty interface.
This basic idea is something I've applied to a file format that stored objects, but where the costs of serializing and deserializing a whole object graph was prohibitive for a request/response round-trip. For any given object path, foo.bar[3].baz, a "program" could be compiled that could be interpreted over the file contents and the answer retrieved. All object paths were available for compilation ahead of time (it was a custom language, long story), so this approach could be far more efficient than any serialization story.
Sounds similar to the Untrusted Deterministic Functions used in exokernels to let userspace do god-knows-what to underlying hardware resources in an efficient but provably safe way, using a correct-by-construction language [1,2].
Another tool in this space is Native Client; any chance it could be adapted to allow an even smaller, safer x86 subset, so that you could give the kernel pre-compiled 'callbacks' that it could execute on, e.g., receiving a network packet, with 0 context switches? Probably not without a lot of work, since NaCl relies a lot on virtual memory for memory isolation, but it's an idea.
Nice links. The UDFs look like a very similar idea to mine, taken further and generalized; I make no claim to originality, it seemed obvious to my compiler-oriented eye.
Oh, I didn't interpret it as such either; just some interesting related work :)
Another peripherally-related idea is Active Messages [1,2]. I see it as the analog to UDFs or other rich executable messages for efficient node-to-node communication in large clusters. It's sort of like RPC if you squint hard enough; you send little executable messages that run on a remote node, and can send you back some data if you want. I guess the point is that having rich executable messages allows you to get around long physical latencies between sender and receiver, as well as more mundane intra-node delays due to userspace/kernelspace context switches.
The same system as the anti-serialization mechanism I described also used an RPC-like mechanism between client web pages and the server. From the JS perspective, it queued a list of commands to execute, with a return value closure passed along to each command. AJAX (pre-JSON days) roundtrips were minimized, because all the commands could be executed sequentially on the server, all results batched up, and returned.
The anti-serialization mechanism also acted as a transactional store - either all instructions succeeded, or the server effectively had no state change (DB transactions etc. were included into a distributed transaction as necessary). No in-memory state. It was a neat architecture (still is, I guess it's probably still in production).
That reminds me of Singularity OS, an experimental OS from Microsoft Research, which has no distinction between kernel and user space, thanks to writing everything in a variant of .NET bytecode.
There have been Java OSes too, with the idea that verifiable bytecode does away with the need for "expensive" memory protection etc. Running everything in one memory space is error-prone, but it's the default way of the monolithic kernel. Having services delegated out to processes with separate memory spaces is architecturally more stable; a single bug in an obscure driver in a less actively maintained corner of the kernel shouldn't lead to a system-wide vulnerability. The approach I mention preserves the ability to keep things separate (with appropriate controls and verification at the boundary); the Singularity / Java OS approach seems rather different.
Yet another variant on the idea was the AS 400. It had no memory protection and didn't need it, since there was no way to address anything that you weren't supposed to.
Given the impressive stability and security record of the AS 400, they had a point.
I'm pretty sure Lisp OSes (you know, on the bytecoded Lisp Machines from companies such as Symbolics and TI) worked like that, and I know the CISC variants of the AS/400 architecture worked like that as well. (I don't know the RISC AS/400 or iSeries that well.)
An interesting aspect of that is what it does to your security model: In the AS/400 world, where applications are compiled to bytecode and then to machine code, the software that compiles to machine code is trusted to never emit dangerous machine code, as there are no further checks on application behavior. In the Lisp Machine world, anyone who can load microcode is God. In Singularity OS, the .Net runtime is effectively the microcode and the same remarks about Lisp Machines apply.
This basic idea is something I've applied to a file format that stored objects, but where the costs of serializing and deserializing a whole object graph was prohibitive for a request/response round-trip. For any given object path, foo.bar[3].baz, a "program" could be compiled that could be interpreted over the file contents and the answer retrieved. All object paths were available for compilation ahead of time (it was a custom language, long story), so this approach could be far more efficient than any serialization story.