That really depends on what kind of software you're running. There's a wide range of portable C/C++ Unix stuff that compiles fine on ARM. If you run a distro like Debian or ubuntu, you can just install the armhf port and have all that stuff.
Where it gets tricky is if it's something written for an environment that itself has no ARM port yet, such as the JVM.
JVM or even Postgres can't run or do not perform well on a cluster, because this type of software takes advantage of shared memory, which is a blocker here.
You can set up replication in Postgres or use some AMPQ for connecting JVMs, but still I would rather buy 3U machine with 4 CPUs and terabytes of ram for XXk$ rather than buy 3U container for ARM servers for XXk$ and spend XXXk$ on engineering work.
In long run having scalable software is win, but most of businesses will do just fine with unscalable software and good backup strategy, which they would need with ARMs anyways.
Where it gets tricky is if it's something written for an environment that itself has no ARM port yet, such as the JVM.