Happens all the time. A colleague of mine has issues because she happened to have worked in a lab previously whose building also happened to house a lab that did nuclear research (even though she never did any herself). Scientists also move around a lot, so it doesn't help if you've worked in/visited/just been to a conference in certain countries that might arouse suspicion... It's a bit of a nightmare actually.