Having had this happen a few times, I'm pretty obsessive about workholding now. It's a shame that a good drill-press vise can cost almost as much as the drill press itself, because it's really important for safety.
You should image-search the term "radial drill" sometime. ;)
You don't need a high-dollar Kurt milling vise for a drill press, and you only need to constrain the part in one degree of freedom (rotation about Z axis). Just put a single fastener in your table and drop the slots in that vise over it. Gravity and quill pressure hold it in Z, the table holds it flat against rotation about X and Y, and you want it to have the ability to slide a bit in X and Y to let the drill find the center of your punch/pilot hole. If the drill catches, the torque will push against the stud and it can't move.
You should image-search the term "radial drill" sometime. ;)