It's a lot less like data hoarding than keeping a separate copy of your social graph. What is an adversary going to do with a list of phone numbers that are known to have signal accounts and nothing else?