It's actually surprisingly hard to intentionally damage most magnetic media (except audio tape and really cheap magstripe cards and you probably aren't going to see either today) with permanent magnet, so one can safely assume, that this is not a problem.