To outsource things step-by-step to lightweight processes seems to be a good incremental way to disentangle. Perhaps one could protect access to resources not initially invented by Emacs packages (Emacs core?) and make Emacs query a separate process, without a package knowing about that, so that it channels all communication to the separate process. Then deprecate directly the resource at some point and offering a more direct way of communicating with the separate process. Finally removing the state in the main process. I think more Emacs internals knowledgeable people would need to judge this idea.
Is there anything in the communication of LSP making the LSP stand out from any other protocol? (I really mean the protocol, not the infrastructure on which it is used.) Other than that it is quite an old idea to have things at the ready running in a separate process. This time it is applied to editors, completion, type checking and other features. In general this simplifies making use of multi-core systems. Watch any Joe Armstrong talk about it. The question is, why it was not thought of before, or perhaps, if it was, then why it was not done before.
In terms of LSP, I don’t think there’s anything particularly novel about it. It’s really only better than other similar protocols in the sense that it is gaining in popularity. I.e., the biggest thing that was missing is consensus on a shared protocol.
In terms of the protocol itself, I suspect it’s actually worse than a lot of other ones.
> I suspect it’s actually worse than a lot of other ones.
Choosing JSON to communicate certainly could have been more wisely decided. Moreover since sending entire file contents across for some queries, what a waste.
It indeed isn't. Some Emacs packages use that approach, and others are well-known to employ Unix utilities to do the legwork. Maven and Gradle both have daemon modes. Unfortunately, all these integrations are custom, so there is no chance for it to take off. Ycmd is also worth mentioning.
What really made the difference is that VS Code uses LSP to integrate languages. Instead of dozens of ad-hoc integrations, there is one unified way to integrate a language now, and Microsoft's backing ensured there is an ecosystem for it from day one.
Not all is rosy though, mind you. LSP uses JSON/RPC after all. Also, we are touching the Microsoft world here, which means that standard adherence is not necessarily a given[0].
Is there anything in the communication of LSP making the LSP stand out from any other protocol? (I really mean the protocol, not the infrastructure on which it is used.) Other than that it is quite an old idea to have things at the ready running in a separate process. This time it is applied to editors, completion, type checking and other features. In general this simplifies making use of multi-core systems. Watch any Joe Armstrong talk about it. The question is, why it was not thought of before, or perhaps, if it was, then why it was not done before.