For a while I wanted to explore new optimization opportunities for ArkScript, and so I wrote a very simple IR for my language ArkScript (lisp like that’s made easy to embed in C++ apps).
I’m now going to improve this IR (to have function calls directly in it, so that I can do some inlining)!
Hm, you seem to be implementing the exact same kind of IR and bytecode that everybody does. There are a lot more possibilities that are rarely mentioned, and some I argue are better. I have an extensive list (/ wannabe tutorial), which I believe all VM writers at least think about, regardless of what actually ends up being implemented: https://gist.github.com/o11c/6b08643335388bbab0228db763f9921...
I will confess to being a strong opponent of stack-based VMs. Admittedly, function calls are the one thing that is notably simpler in them compared to register-based machines.
Instead of 32 bits per instruction, have you thought about 16 bits per instruction then doing something special for the rare instruction that needs more (either a flag for adjacent "extra data", or a function-level table to index, or even a special register)? Especially in a stack machine you're likely to have a lot of instructions with no argument at all.
You're doing label-based IR, but block-based IR seems to be more popular these days (though that is mostly with SSA). Regardless, think about the case where optimization leads to empty blocks / no code between adjacent labels (it's probably fine for your current approach, but only because you currently can't have large functions at all). Also, it doesn't make sense to use `unordered_map` when `vector` will do.
