I'm not an astrophysicist, but I was part of a D&D group with one a few years ago and the topic came up (outside the game). In practice the speeds fall into two fairly tight clusters (asteroids and comets), but you don't even need that to justify focusing on mass. There's a hard lower bound on everything, and also a hard upper bound on any object that is part of our solar system, and it works out to at most a factor of 40ish in kinetic energy between the slowest and fastest possible impacts. The masses of objects of interest have a much wider range.