It's not an imperial unit, but it's not an SI unit either. One glaring example of it's inappropriateness is that the 'calorie' varies depending on what you're measuring (as you say, sometimes a gram, sometimes a kilogram). The SI unit is the joule.

I'll accept that. The calorie / Calorie / Kilocalorie confusion is pretty annoying.

Apparently EU food labeling may be in either kcal or kJ (at ~4.2 kJ/kcal).

Australian labelling must be in kJ (but it will often have the kcal amount listed as well)