Both can be made to work quite well for various ANN implementations. Currently GPGPU is a lot more popular because it requires far less hardware knowledge to get going, which translates into much quicker development cycles. For example, the boom of deep learning which has been going on since ~2002 has been largely couched in the massively parallel training of neural networks on GPUs. Some clever folks have written libraries like Theano, which allows you to create computation graphs in python that can be compiled, targeting either CPU or GPU.
In principle FPGAs can be an extremely attractive option, because a well designed implementation could run very fast as scale very well, given that their massively parallel architecture lends itself to the parallel nature of ANNs. A few groups have gone to lengths to implement, for example, a Restricted Boltzmann Machine architecture directly on FPGA, and in raw benchmarks it works well. However, you start to run into practical limitations if you try to apply those archs to real-world problems, because all but the most expensive FPGAs have trouble fitting very large neural net architectures on-chip. Furthermore, the data required to train the nets is often too big to fit in RAM on the chips, meaning you're left with network or SD-card type approaches, which can be cumbersome and slow. I can recall at least one instance where the design explicitly addressed this issue by designing the architecture to be distributed over many hardware linked FPGAs, but even then you still run into the usual parallelism and concurrency issues: you need to balance parallelism overhead against speedups, you run into synchronization issues, and things can become non-deterministic and very difficult to debug.
The bottom line is that it doesn't look like neural network FPGAs or ASICs are there yet. I'm sure there will be more exploration of this space in the future. As it stands right now, neural network training is dominated by GPGPU implementations.
In principle FPGAs can be an extremely attractive option, because a well designed implementation could run very fast as scale very well, given that their massively parallel architecture lends itself to the parallel nature of ANNs. A few groups have gone to lengths to implement, for example, a Restricted Boltzmann Machine architecture directly on FPGA, and in raw benchmarks it works well. However, you start to run into practical limitations if you try to apply those archs to real-world problems, because all but the most expensive FPGAs have trouble fitting very large neural net architectures on-chip. Furthermore, the data required to train the nets is often too big to fit in RAM on the chips, meaning you're left with network or SD-card type approaches, which can be cumbersome and slow. I can recall at least one instance where the design explicitly addressed this issue by designing the architecture to be distributed over many hardware linked FPGAs, but even then you still run into the usual parallelism and concurrency issues: you need to balance parallelism overhead against speedups, you run into synchronization issues, and things can become non-deterministic and very difficult to debug.
The bottom line is that it doesn't look like neural network FPGAs or ASICs are there yet. I'm sure there will be more exploration of this space in the future. As it stands right now, neural network training is dominated by GPGPU implementations.