With QAM you're more thinking about the error vector magnitude (EVM), rather then the signal-to-noise ratio. From the EVM you get a bit error rate, which determines maximum sensible data speeds. How high QAM will go is also a strong function of what frequency band: at low frequency, I've seen 512-QAM and higher, while at >100GHz they're struggeling for 32-QAM