The Linux philosophy is important to me in a desktop, but I don't think I'd really care about it on a phone. If it were eventually to be both, then I would, but I never really believed in that goal. If I'm going to use a computer in desktop mode, I'd rather just have a proper desktop, with hardware that is unrestrained by the smaller form factor.
You can use a Librem 5 in this exact way. I do it - it's fast enough to run an IDE and do some programming (as long as the thing you work on compiles in reasonable time), check e-mail, IM and social media, do some light browsing etc. I have GIMP, Inkscape, Audacity and LMMS there, which together with Qt Creator are pretty much all the tools I need for my gamedev work - and now I can do that on my phone attached to a screen, keyboard and mouse. Jamulus, Mumble, mpd, even some lighter games and emulators - it all works, straight from Debian repos. It's pretty fun.
I have even ran a full Plasma Desktop on an external screen this way - and it worked surprisingly well even despite of GPU acceleration not being available there because of a bug.
