There's a really good explanation in the book "A Bug Hunters Diary".

It used an old version of vlc and a buffer overflow vulnerability to get code execution.