I've found that my tolerance for spicy depends entirely on the spice. Chili peppers? Intolerable to me. Even "mild" dishes are too uncomfortable, and anything spicier is impossible for me to eat.

On the other hand, horseradish or wasabi is no problem even in high concentrations. Black pepper is no problem as well.

It seems to me that there must be a biochemical reason for the differences.