In addition to this excellent advice, I'll just add that my experience is that a 2GB CUDA compatible card is (just) enough to complete this course. Although 4GB makes life much easier
This is what I have and it's been holding up great for everything I've wanted to do till now. Core i3 6100, Nvidia GTX 950 2GB, 8GB DDR4 RAM. Did not try running a neural net on the GPU, but it should do the job. The whole thing - http://lonesword.in/hardware/2015/11/10/Assembled-a-computer... - including the cheap table and chair cost me around 60K indian rupees which roughly translates to 1000 USD - but that's because computer components are around 30-40% more expensive in India than in the US.
A used Dell workstation off eBay plus a GTX 1070 will get you exceptional performance for under $1000. If you want to spend less, a used GTX 980 is also a good option.
I just set up CUDA and cudnn for python3 and tensorflow last night on my machine - with that same card. That card should have 640 cuda cores; every little bit counts, imho. I'm not really sure what the difference will be compared to what I was doing before, but anything has to be better than the quad-core cpu I'm currently using (some older AMD thing).
