> if your source is low volume and your output is cranked up you get a lot of...distortion? Interference? Artifacts?

Mostly noise because you're amplifying it as well, and possibly artifacts because material converted at too low levels (which is the case of old CD rips that weren't normalized before encoding) wouldn't use all the available headroom, therefore losing one bit.