Well yes the term tea is used "in the wrong way" a lot and one could argue that it is common enough that it's the right usage these days (language evolves etc.). But technically tea is only from camelia sinensis varieties. So the "herbal teas" should technically be called something else like tisane.
Chinese call herbal teas “liang cha”; the argument that “herbal tea” should not be called tea is a somewhat modern English notion. Other languages freely use tea/cha for infusions of many plants.
Heck, why not argue that any “tisane” not made from barley isn’t a real tisane.
