I don't see a lot of options how you should optimize that, besides using allocation free code. .NET (Core) did a lot of work into this direction for Kestrel HTTP server already (Span, ref structs). It's still a managed platform with a garbage collector, but I don't think games have such low-latency requirements, that can't be handled with the standard garbage collector. A few milliseconds are usually fine for games, even with 120Hz the time between two frames is still 8ms. And garbage collection can be suspended during critical phases.
