> The privilege is a result of a system. You cannot say x people have privilege

This is a distinction without a difference. The Principle of Charity applies and we can easily understand that "You have privilege X" is being used to mean "You are accorded privilege X by the dominant social system in which we both exist".