Crossplane maintainer here -- this has come up a number of times in the community, but ultimately we would prefer for the system to be able to handle an arbitrary number of CRDs, whether they are being used or not (as evidenced by the work outlined in this post). Moving to filtering CRDs that a provider installs can lead to a confusing experience where the presence of a given package does not mean the same thing across clusters, which also complicates the dependency model Crossplane packages implement. We view Composition + RBAC as the mechanisms to define what a given entity can access (i.e. the API line).
I can actually say that not supporting this really does hinder crossplane adoption. At work we operate large shared kubernetes clusters. A team attempted to use crossplane and we immediately had to remove it from our clusters immediately because it made kubectl and a variety of other tools completely unusable due to all the rate limiting and how frequently it refreshes cached discovery data. They wanted to use crossplane for like 10-20 of its supported object types. Instead, they had to actually run crossplane inside of a vcluster because there is no option to filter the number of CRDs it creates.
So while I can get behind this sentiment philosophically, until something changes upstream in kubernetes, this makes it really difficult to use crossplane in a cluster used for anything else and it probably makes sense to offer a workaround until then. Also, in practice, any security conscious users running crossplane in production are probably going to give it AWS credentials scoped to only the resources they want to allow it to manage, so even if you do install all of the CRDs in the cluster, 90% of them won't work due to their AWS credentials anyway.
