That's why Signal only stores your phone number (and when you last connected) - they know nothing about your real identity, so they can't link it back to you.