I see alot of people saying how certain configs are missing/better/not tested. So what about a "crowdsourced benchmark" instead?
Maybe a few elaborate scenarios are posted and people can simply submit their best setup/config/code to be benchmarked. I imagine devs would improve on it over time and eventually the most optimized would surface?
We are doing precisely that: if there are any mistakes or misunderstandings about the production-level best practices, we are accepting pull requests on our github (set up with all these tests, in case you want to run them yourself) to fix them, rerun them, and then report our updated findings.
Maybe a few elaborate scenarios are posted and people can simply submit their best setup/config/code to be benchmarked. I imagine devs would improve on it over time and eventually the most optimized would surface?