You know, that's the problem. There is basically no reason why this is so hard.
Many security features could just be enabled by default by major distributions with hardly any downside. You don't even have to look at grsecurity. Just using pie binaries to enable proper ASLR would be a start.
I'm not well versed enough to understand whether "Just using pie binaries to enable proper ASLR" is included, but the chart does show green against various things mentioning ASLR. It looks like specific packages are built with PIE, too.