I have a slight physiological twitch-like reaction to the feeling of chalk on my fingertips. It bothered me so much during my learning years that I did my best to avoid coming to a board at all. If anyone knows a correct term for this phenomenon, I'd be glad to read more about it.
I don't know if there's a name, but you're not alone. I f$%*ing hate chalk, so much that I started avoiding math classes at my school. Both the sound it makes and the feeling of it as you mention. Just thinking about it makes my skin crawl.
We're not crazy though. Just look at what chalk looks like under the microscope to see what your fingers are picking up on:
