Relying on it is not the problem. Treating it (or "date of birth" or "mother's maiden name") as a secret for use in authentication is a big problem. These things are not secret, and having me say mine does not prove that you're talking to me.
In my (shared) office, everyone knew each other's last 4 SSN digits, because whenever on the phone to some random customer service rep, we had to give them to "authenticate".
It would be just fine to rely on SSN as an identifier, even to a much larger scale as USA does now, if only it would be clearly assumed that this number isn't secret.
