Not every human is created equal when it comes to ear and sound processing. I have met people with sub 20Hz hearing, and people with ears so sharp, they were able to pick a single wrong note from a single instrument while watching a recording of a symphony orchestra (I played together with them).
MP3@320kbps CBR and AAC@256kbps are pretty good for normal listening, but if you have the hardware to render, lossless formats creates a richer soundstage. I have an amplifier which can render it, and I'm listening music with it for 30 years now, so I can hear the difference.
At the end of the day, if your audio pipeline can render the differential residue between MP3@320kbps and FLAC, you can hear it.
Now, you can say that "are you attentive enough to perceive such difference", I'm not listening that intently 75% of the time, but it pays off when I put some time aside to listen to my favorite album for the sake of listening it.
By far the main source of degradation in any typical analog audio path is going to be transducers (microphones, speakers, phono cartridges, tape heads) and inferior media (tape, etc.). The vast majority of modern amplifiers, and high end older amplifiers, are extremely transparent with good margins beyond typical human hearing; any issues like noise, harmonic distortion, uneven response, inadequate damping factor, etc. introduced at the amplifier would typically be masked by quirks of the speakers, and revealed only by measuring upstream of the speakers.
For active listening, the gain going from 320 MP3 / 256 AAC to lossless is beyond minuscule. What you are hearing is far more likely to be placebo.