I mean, sure, in the long term. But there is a long, long list of stuff that the military uses that I'm more worried about than DU. It's essentially like Tungsten, Lead, or Mercury.
The problem more or less unique to DU munitions is that it gives off tiny aerosol particles when it hits its target which can contaminate a large area and are very easy to breathe in for a long time afterwards. And breathing in these particles are linked to an increasing number of health issues.