> but once you activated all the quality-of-life features and added a bunch of third-party packages, there was a noticeable performance impact.
I'm not sure I understand what you mean there. Unless those third-party packages are scheduling tasks on the event loop then they should not be impacting performance of your other operations. Just having lots of things installed and activated doesn't affect performance.
With respect, I think you're confused. But even if I'm right, that's OK, software is complicated, and it took me a while to get these things straight.
Multiple installed emacs packages do not execute their code simultaneously unless they are using something like run-with-timer to schedule tasks on the event loop. Can you give me an example of an emacs package that causes custom elisp to be "running in your buffer" continuously?
For example, org-mode does many complicated things, but only when you ask it to -- i.e. when you issue one of its keybindings, or call one of its functions. Then, you enter a blocking ("synchronous") function call: the main thread is executing the org-mode code and nothing else until it's done. So, if things happen as I'm describing, you can see that it doesn't matter how many packages you have installed: only one of them is taxing the CPU at any one time, and they are not all queuing up work and causing the event loop to get bogged down.
Similarly, swiper, ivy, counsel, magit -- they only do things when you ask them to.
To put this another way: in general, your installed packages are not active simultaneously. Therefore, the performance of your editing operations is not affected by how many packages you use.
> Multiple installed emacs packages do not execute their code simultaneously unless they are using something like run-with-timer to schedule tasks on the event loop.
It's true that few things run in the background constantly. You can check for that easily, with M-x list-timers. On my work laptop, my most feature-packed Emacs instance, I currently have three timers active (and 11 waiting for a trigger). Those three are:
Repeat Function
5.0 auto-revert-buffers
60.0 ac-clear-variables-every-minute
3600.0 url-cookie-write-file
So there's only one frequently updating timer running, that was put there by (I think) lsp-mode.
Of the inactive timers, there are a bunch on a really short trigger - for features like highlighting indentation guides, highlighting parentheses, etc. But these are the ones that wake up when you do just about anything in a relevant buffer - along with various hooks being fired.
Because it's the hooks that seem to be missing in your current understanding.
I just switched to a random buffer, and in it, I have 18 functions in post-command-hook, and 2 functions in post-self-insert-hook. That's 20 functions to execute on pretty much every single key press, and all of them come from optional features, both built-in and from additional packages. Syntax highlighting (font-lock-mode), error checking (flycheck), documentation helpers (eldoc), LSP stuff, snippet expansion, etc.
All these functions are usually fast (plenty of them just reschedule aforementioned inactive timers), but every now and then, one of them gets some heavier workload. And you notice a stutter. Perhaps inserting a character took more milliseconds than you're used to, because it caused font-lock to re-render the entire screen. Perhaps autocompletion triggered some expensive checks. Or perhaps there's just too many badly-written functions firing, and it added up to a noticeable delay.
For me, the three most common cases of annoying little delays were autocompletion (in Elisp, and Common Lisp via SLIME), font-lock and org mode folding. The latter two were an issue when working with large org mode files.
Native compilation improved Elisp performance across the board, and all but eliminated these issues for me. It won't help you if you put a badly written function in a frequently firing hook, but it does push some of the more expensive computations below the threshold of visibility, and makes it less likely that a lot of functions in a hook will add up to a noticeable delay.
Similarly, I wanted to check what was taking up time on redoing org-agenda, profiler says the majority of the time is spent doing `kill-all-local-variables`
These are all third-party packages, have nothing to do with org-mode. Pretty much any globalized minor mode you enable will add some constant time to switching into fundamental-mode, and it all adds up.
(Also, half the time when I run the profiler, the culprit ends up being evil-mode. Unfortunately it's so useful …)
Wow, that's an interesting result, I never noticed it directly - I've seen -cmhh functions before, but never bothered to check what they are before.
As it turns out, these are automatically created by `define-globalized-minor-mode' macro - per docstring, it's used to define a global mode for buffer-local minor modes. I.e. that's how you get all these `global-XXX-mode` functions that turn on XXX-mode in every (relevant) buffer.
One of the things this macro does is create a `global-XXX-cmhh' function, and attaches it to `change-major-mode-hook', which is invoked by `kill-all-local-variables' whenever buffer's mode is changed. I imagine that, as part of generating org mode's agenda, files get opened and modes get switched a lot. Curiously, it also seems that (digging into `kill-all-local-variables' source in C), each call to it forces mode-line redisplay, so further computation may be caused by whatever is in your mode line (which can also be nontrivial). I haven't checked that though, maybe it redisplays mode line only once.
Thanks for bringing this up, it's another place where pretty much every mode stuff some hook, that I didn't know about :).
EDIT: curiously enough, I just profiled my agenda - both doing from scratch and redoing it - and `kill-all-local-variables' isn't even on its radar. I wonder what runs in your hooks (and/or modeline) then. The result from refreshing an existing agenda was roughly what I expected. Building it anew, after killing all open org buffers, was interesting. In my case, 69% of total time was spent opening files (`find-file-noselect'), before even doing any mode switching. In that, `projectile-find-file-hook-function' accounted for 50% of total agenda building time, half of that spent updating the mode line!
From what I see, it's just trying to determine the project to which a file belongs. I'll probably look for an option to make projectile stop being interested in files that aren't in the "known projects" subtrees without explicitly instructing it so.
Ah, thanks for explaining that so clearly and thoroughly. You're right that I was forgetting hooks. (And I learned something else from what you wrote -- that often a library will have it's triggered function merely enqueue a task. Of course that makes sense but I'd forgotten that in an emacs context (whereas I might have thought of it in a backend web dev context)). Two questions/comments:
1. I had it in my head that emacs arranged for font lock to be done in a separate thread? (Also, why is it called "lock"? I just use that word cos the emacs docs and code do)
2. Isn't org mode folding/unfolding just a blocking call that happens when you ask it to?
> emacs arranged for font lock to be done in a separate thread
I'm not sure, but I don't think so. There's this thing called `jit-lock-mode' (check out the docstring of the function named jit-lock-mode for details) that makes Emacs fontify only visible parts of the buffer when triggered by redisplay code (in C core), + some extra fontification of the invisible parts on idle timer. But I don't believe it actually happens on a separate thread, given that redisplay can run arbitrary user code, e.g. through 'display property attached to a piece of text in a visible buffer.
> Also, why is it called "lock"? I just use that word cos the emacs docs and code do
So do I. I did a little googling, but couldn't find any definitive answer. Most likely explanation[0] seems to be that "lock" here means the fontification spec is attached to the text and updated automatically, vs. being refreshed globally on user request.
> Isn't org mode folding/unfolding just a blocking call that happens when you ask it to?
It is, but it's also something I do very frequently in quick succession. I'll usually press S-Tab in quick succession to perform global visibility cycling, because I often want to take a quick look at the outline of my file, and then get back to editing where I was. If it takes more than a fraction of a second, it's distracting for that use case.
It certainly might have been perceived that way. IIRC, Font-lock mode included logic to defer the heavier execution of fontification until a certain amount of idle time accrued (particularly helpful from a UX perspective as people are typing there can often be dramatic shifts in fontification, and it isn't worth slowing them down just to make a change that will disappear at the next keystroke). There are also tricks in it to do partial fontification. I believe this is now handled in a sub-module called jit-lock-mode (Just-in-time Lock Mode).
However, font-lock-mode never executed in a separate preemptive thread.
> Multiple installed emacs packages do not execute their code simultaneously unless they are using something like run-with-timer to schedule tasks on the event loop.
Yes, that's accurate, but that's the whole problem. The code executes sequentially, so each custom execution adds latency. It's not unusual to have elisp from several packages executing with every keypress.
I'm not sure I understand what you mean there. Unless those third-party packages are scheduling tasks on the event loop then they should not be impacting performance of your other operations. Just having lots of things installed and activated doesn't affect performance.