GPUs effectively have general-purpose compute now and they can DMA bytes into and out of system memory. At a basic level, sneaky on-GPU code run by an attacker could grab pixels off your screen (text from your emails, etc), but it's also possible for the GPU-based attack code to harvest data from system memory or even mess with data in order to attack a process running on the CPU.
Graphics APIs are defined and implemented in memory-unsafe languages on the CPU side, C/C++, and will read from and write to memory based on data and pointers you pass to them without any provisions for untrusted data. -> instant pwnage
(Not to say GPU code execution is safe, but you wouldn't need it to ruin the users day)