It's more useful in an evolutionary sense to look at a banana and see that it's yellow, than to feel distinctly that your red and green cone cells are each being stimulated separately.

The combination happens in our brain because it's transforming those separate signals into useful information.