Milk Allergy can be dangerous. It depends of the level of sensitiveness. For example a baby can be vaccinated by Glaxo's Infanrix that not say explicitly that contains milk protein and he/she will have rush in the skin, fever, cry all the day and make blood stools. And this is just a small symptom. In the worst case it can have an anaphylactic reaction.
So... it can be really serious.