And the pipeline is able to convert it to RGB fine, so isn't the information available to use? Why would it be a challenge passing along that information?
The information is there (sometimes... it's amazing how many times you have to make a "reasonable" guess when decoding images/video because metadata is missing). Passing it along is feasible. But it does complicate things (especially the implementation) pretty quickly.
A lot of Y'CbCr -> RGB converters actually disagree with each other. They're all close enough that casual users don't notice or care about the small discrepancies.
