That's a narrow definition of "commodity" -- the special networks cost less than the same speed of Ethernet, and Intel server chips (non-phi) aren't that different from desktop CPUs.
If you look through the archives of the beowulf mailing list, occasionally someone makes the argument you're making, and few people agree with it.
There is no 'same speed of Ethernet' for infiniband or omnipath or aries, etc. There is more to these networks than throughput, and the switches approach a million dollars apiece.
The rest of the non-phi/non-tesla hardware is pretty much off the shelf, but the interconnect is one of the two distinguishing features of a supercomputing-class cluster; the other is high-performance shared storage (which of course requires the interconnect to function).
It's a shame feel like I need to. There's no world where high-speed interconnects are as cheap as ethernet, nor is there a world where it is appropriate to replace them with ethernet. Congratulations on your successes but they're not really relevant to the accuracy of your post.
If you look through the archives of the beowulf mailing list, occasionally someone makes the argument you're making, and few people agree with it.