I've heard people say it's racist for white people to say "y'all" too, since black people say "y'all" therefore it's cultural appropriation.

Nevermind that "y'all" is common among white Southerners (where do you think African Americans picked it up from?). Why let facts get in the way of your attempts to control people?