Emacs is definitely capable of multithreading. Still using a single thread might actually be a boon because it simplifies the programming model. Right now, Elisp code can work under the assumption that there is only one thread accessing editor state at the same time.
Disentangling this is probably going to be difficult, but the Emacs folks are probably smart enough to come up with something that matches with the spirit of Emacs. Libuv and callbacks might be a way, but cooperative scheduling of user threads on backing threads together with structured concurrency, like Java's Project Loom is attempting, are another way.
The emergence of the LSP protocol is a very promising development in the architecture of editors and IDEs. Many things can and maybe should be passed off to background processes to handle, both to expand the feature set of every editor out there, and to increase speed and stability.
To outsource things step-by-step to lightweight processes seems to be a good incremental way to disentangle. Perhaps one could protect access to resources not initially invented by Emacs packages (Emacs core?) and make Emacs query a separate process, without a package knowing about that, so that it channels all communication to the separate process. Then deprecate directly the resource at some point and offering a more direct way of communicating with the separate process. Finally removing the state in the main process. I think more Emacs internals knowledgeable people would need to judge this idea.
Is there anything in the communication of LSP making the LSP stand out from any other protocol? (I really mean the protocol, not the infrastructure on which it is used.) Other than that it is quite an old idea to have things at the ready running in a separate process. This time it is applied to editors, completion, type checking and other features. In general this simplifies making use of multi-core systems. Watch any Joe Armstrong talk about it. The question is, why it was not thought of before, or perhaps, if it was, then why it was not done before.
In terms of LSP, I don’t think there’s anything particularly novel about it. It’s really only better than other similar protocols in the sense that it is gaining in popularity. I.e., the biggest thing that was missing is consensus on a shared protocol.
In terms of the protocol itself, I suspect it’s actually worse than a lot of other ones.
> I suspect it’s actually worse than a lot of other ones.
Choosing JSON to communicate certainly could have been more wisely decided. Moreover since sending entire file contents across for some queries, what a waste.
It indeed isn't. Some Emacs packages use that approach, and others are well-known to employ Unix utilities to do the legwork. Maven and Gradle both have daemon modes. Unfortunately, all these integrations are custom, so there is no chance for it to take off. Ycmd is also worth mentioning.
What really made the difference is that VS Code uses LSP to integrate languages. Instead of dozens of ad-hoc integrations, there is one unified way to integrate a language now, and Microsoft's backing ensured there is an ecosystem for it from day one.
Not all is rosy though, mind you. LSP uses JSON/RPC after all. Also, we are touching the Microsoft world here, which means that standard adherence is not necessarily a given[0].
Indeed. It does not need precisely multithreading, but at least asynchrony, which is to allow side tasks to not block the main function of editing text. I have low familiarity with emacs internals, but I can assume there is an event loop, which is a basic form of this.
Perhaps this needs be used more, or there is another mechanism to be developed allowing tasks to be interrupt/resumed in priority service to the editing functions. A key example may be mode line refresh: this certainly must happen on the main thread, but it ought not block other more important items.
With a few exceptions, I don't think moving stuff into the background promotes speed and stability. It makes it way easier to get away with reductions to both.
Disentangling this is probably going to be difficult, but the Emacs folks are probably smart enough to come up with something that matches with the spirit of Emacs. Libuv and callbacks might be a way, but cooperative scheduling of user threads on backing threads together with structured concurrency, like Java's Project Loom is attempting, are another way.
The emergence of the LSP protocol is a very promising development in the architecture of editors and IDEs. Many things can and maybe should be passed off to background processes to handle, both to expand the feature set of every editor out there, and to increase speed and stability.