I have never had problems distributing binary builds for Linux. There are a number of system libraries that are backwards compatible: glibc, asound, libGL, etc. - test on the oldest you want to support and any other libs you just ship yourself. And that would not be different with only one distro either unless you only care about one release of that distro.
Unless you want to ship something, troubleshoot something, or figure out how something works without hours of googling.