Well your signal client performs SGX remote attestation before sending any contact data to ensure that the server codebase matches a valid release. So if they're not running the published source, your client will refuse to share your contact information and social graph. Note that messages are e2e encrypted on the client side, so they don't enter into it.