You mean startup time without using the server and client setup is too slow, right? (That's easily solved by, well, using the server and client setup, of course.)
I find that in actual operation Emacs is faster and that's the reason I switched from Vim to Emacs. Even after using all the tricks in Vim's :help tex-slow, Vim still lagged when I scrolled or even typed into LaTeX files (unless I turned off syntax highlighting), and I'm not such a great typist. Emacs is perfectly snappy.
Ah, I'm not a Latex person. In my experience, vim works fine, even on very large files, unless you combine syntax highlighting with very long lines, in which case you're going to suffer.
But yes, I admit that I find silly that I should need a 'server' for a text editor. Not to mention that I remote into various VMs regularly which would also need to get their own server/client setup if I wanted a homogeneous working environment.
I use emacs in a work environment that is primarily Vim and where we end up logging in to a lot of machines to do development on, and the two things that turn heads in my emacs setups are my fluent and flexible use of multiple windows and frames, and the fact that I'm always using my local emacs and I use Tramp to remote in to the machines, so I don't have to screw around with .vimrc files on half-a-dozen machines. Both are stock emacs.
It sounds like it's only practical when you know you want to edit things in the first place, as opposed to "it doesn't work, let's restart the service in my existing SSH session and have have a look of the log" sysadmin-style-workflow.
I hate to disappoint, but it really works quite well. Not quite perfectly, but quite well (and most of the trouble I encounter comes from my use of ControlMaster on the SSH side). No, you don't any sort of weird heroics to use it.
I only found Vim syntax highlighting annoyingly slow on TeX, LaTeX and Pandoc Markdown (which supports embedded TeX math formulas), so maybe the TeX syntax highlighter is the single source of slowness. (Also, there is a tex-slow section of the Vim manual, but I think there aren't sections like that for other languages...)
But moving to Emacs had many other benefits, so I'm definitely glad I did it. The LaTeX is just what made me look for other editors, being the one thing that really bothered me about Vim.
And about the remote VMs: as user jerf said, you only run one Emacs, on a local machine, and use Tramp [1] to edit remote files as if they were local. It's great!
I find that in actual operation Emacs is faster and that's the reason I switched from Vim to Emacs. Even after using all the tricks in Vim's :help tex-slow, Vim still lagged when I scrolled or even typed into LaTeX files (unless I turned off syntax highlighting), and I'm not such a great typist. Emacs is perfectly snappy.