Why limit yourself? Just run everything in ring 0. No kernel calls overhead at all. Lightning-fast network stack. Use a unikernel, run your trusted code only, don't store secrets.
Beyond game servers, this could be the mode of operation of e.g. compute cluster nodes, relying on external firewall for security, and running zero untrusted code. Of course they won't care about meltdown or spectre either.
I’ve heard some rumblings that this is what some HFT shops were doing years ago to minimize latency and to reduce context switches. Because they already run bare metal and don’t share anything besides a physical facility with their competitors (if even that) the attack vectors are not the same as those running on shared machines either.
Beyond game servers, this could be the mode of operation of e.g. compute cluster nodes, relying on external firewall for security, and running zero untrusted code. Of course they won't care about meltdown or spectre either.