Because VS has limitations on the amount of memory it can use and although they have been working on this, there is still functionality which is highly single threaded whereas Rider tends to offload most of the functionality from the UI thread into background processes.
The memory limitations can be partially kicked down the road by setting /largeaddressaware on devenv.exe. Ofc it's still beyond frustrating when you have a large solution with many component projects. I personally ram against the 4gb ceiling constantly.
I see, this means Rider is faster than the VS on many-core machines with huge amounts of memory. That's why it isn't going to be fast for me - I only use laptops, with 4 GiB RAM and dual cores each.