Unsafe C# is still C# though. Also C# has a lot more control over memory than Java for example, so you don't actually need to use unsafe to be fast. Or are you trying to say that C# is only fast when using unsafe?
Likely just that the fastest implementations in the benchmarks game are using those features and so aren't really a good reflection of the language as it is normally used. This is a problem for any language on the list, really; the fastest implementations are probably not going to reflect idiomatic coding practices.
Here are naive un-optimised single-thread programs transliterated line-by-line literal style into different programming languages from the same original.