This has to do with the symmetry of the input to translations, that allows a convolutional neural network. But neural networks need not be convolutional.