I imagine you would do what erlang does for you, just manually - isolate individual server instances, do the update, then make them public again.
Say for example you are updating code in your service's web-tier. Have your LBs not send any more new traffic to the server instance, wait some reasonable length of time until existing connections are completed, deploy, restart, give LBs the A-OK to start traffic up again. Repeat until all web-tier instances are updated.
Ok but what if that server you isolated was holding a long run process.
Yes for a web service that serves quick http request and responses you could do that. Not everything is short lived request and response messages. Some sessions and processes are long lived. So you have a socket open and data streaming into it, it is not easy to isolate it. You could say send it a message that says -- start isolating.
How do you hotswap with data held in the stack or the heap? Even more if two part of your code need updated instance data, how does it ensure that update is synchronous or happens in the right order? In Java they have a $transformer() method. Ok how do you regulate the order in which $transformer() gets called. Otherwise a new version of one instance will call an old version of another.
I am not saying it is impossible to do it, there might be a way, but it is just usually working against the framework and against the default setup of the system.
Yes, it can get very complicated trying to handle all the edge cases as you described. I would guess doing it the same way erlang does it is actually the simplest: let it crash.
This gets into coding mindset, my impression is that most erlang programmers expect their code to crash whereas most other languages seem to lend themselves to expecting the program _not_ to crash.
I am not very well versed in erlang but my readings so far imply that the graceful handling of crashes is really where erlang/OTP shines. Regardless of the framework, I would say it comes down to proper queuing, being able to safely retry work, and to some extent having some smarts on the clients.
As for C or C++ it is platform specific, but shared objects and DLLs have been runtime loadable/unloadable for as long as I can remember. Building a hot swap feature on top of it would be challenging but doable.
The tricky part is transferring state between hot reloads, and that's part of what Erlang was designed around. State has to be passed in and owned by the callee. And if the format that state is stored in changes, you need to handle that translation in the new version.
It's definitely doable in C and C++. The question is whether it is easier to re-implement a bunch of what Erlang does for you, or to just use something that supports it directly.
