Setting aside any impracticalities of doing this, I've always wondered what an operating system would look like if it was tightly integrated into a specific virtual machine. Something like Java or Racket (or blub if you would like), to the point where only languages that ran on those platforms could run on the operating system.
One of the biggest issues people bring up when someone asks "Why can't the OS handle garbage?" is that the OS doesn't know what is garbage and what isn't. But if you only ran languages that ran on the OSes virtual machine, then it would know the memory model. Java and Racket both have histories of languages being built on top of them, so if you don't like the base language you could create your own.
I'm not saying this would be practical or useful (other than maybe as a proof of concept) but it could be fun!
So there is bare metal Erlang on the Grisp boards and that's usually a virtual machine. Haven't dug into that at all but seems cool: https://www.grisp.org/
The Nerves project is perhaps a more pragmatic approach that is in production use that I really like which uses a fairly light Linux to get the VM up and running: https://www.nerves-project.org/
With Nerves you mostly write Elixir or Erlang to build your device functionality.
Ah man I love Erlang and might have to look into that first board some more. I may be able to port one of my more ambitious projects to it depending on what all it supports.
An operating system which only runs code written in a safe language (without escape hatches) would be useful to enforce capability-based security at a finer level than processes. If you sandbox a program, you could choose which individual functions/classes/modules have access to the filesystem and network by choosing which ones to pass in an object with access to the filesystem. And the program would have no way other than these objects to make raw syscalls (much like you can't call fopen in browser JavaScript).
But from what I heard, Java tried SecurityManager at the language level and it failed. So I'm not sure. Also I haven't researched capability-based security deeply, so I'm not an expert in this field.
When Java was just breaking out, before 1.0, some people proposed changes for capability security, but it didn't get anywhere. Java went with stack inspection instead.
Worth noting that a language-based OS is not necessary for capabilities -- they were invented in the context of the usual hardware memory-protection in the 1960s.
These are for traditional, stack-based JVM bytecode. But as I understand after having spent way too much time lately poking around and modifying .smali disassembly of other people's apps, Dalvik (Android JVM) bytecode was initially designed to map more-or-less directly to actual CPU registers to be easily JIT-able.
But then I don't think garbage collection can be done purely in hardware. You still have to have something resembling a runtime, and it has to be written in a lower level language.
One of the biggest issues people bring up when someone asks "Why can't the OS handle garbage?" is that the OS doesn't know what is garbage and what isn't. But if you only ran languages that ran on the OSes virtual machine, then it would know the memory model. Java and Racket both have histories of languages being built on top of them, so if you don't like the base language you could create your own.
I'm not saying this would be practical or useful (other than maybe as a proof of concept) but it could be fun!