The other reason not to run Linux is because the ESP32's a bit underpowered to run Linux (lacks an MMU). Slightly more powerful hardware (eg Raspberry Pi Zero) would do better, and get you a working device far sooner, which would be great for testing out some of the other components while waiting for getting the software written.
Embedded Linux (eg any distro using busybox for glibc) is (imo) the appropriate stack to use for this given how cheap a microcontroller with an MMU is these days, and once you get rid of all the bloat, is capable of very quick boot times (availability), and if the device is off (and not just sleeping), then its battery life is similarly extended. (Don't let the poor battery life of Android phones let you believe Linux isn't capable of long battery life. Smartphones don't actually get to sleep while the screen is off.)
The ESP32 is getting replaced with a better microcontroller (it's down to either an upcoming STM32 ultra-low power or the upcoming Ambiq Apollo4 depending on how well they perform in real life), but ESP32 was what we started with given familiarity and availability.
Remember, the device can sometimes have enough time to sleep BETWEEN keystrokes while typing. (The microcontroller can sleep then wake and power up peripherals with only a tiny, imperceptible delay.) This isn't your typical Linux setup. FreeRTOS is plenty. For the limited things it does, it wants to be a no-compromise device. There is no on button. There is no off button. You grab the device, it has an image on the display because it's e-ink and you start typing. Assuming you've previously connected to a remote device the system will have done what's necessary to keep that connection alive (invisibly waking as needed). Your keystrokes can go through right away to the remote connection. Your results will hit the screen right away too. The device is always on and it is always off, if that makes sense. There's more to it--bits of stuff and bother and implementation details--but I hope this gives you the idea.
Embedded Linux (eg any distro using busybox for glibc) is (imo) the appropriate stack to use for this given how cheap a microcontroller with an MMU is these days, and once you get rid of all the bloat, is capable of very quick boot times (availability), and if the device is off (and not just sleeping), then its battery life is similarly extended. (Don't let the poor battery life of Android phones let you believe Linux isn't capable of long battery life. Smartphones don't actually get to sleep while the screen is off.)