The point of end-to-end encrypted messaging is not having to care about what the server is running, which is why the threat models for most academic cryptographic research on these things is "assume a compromised server", and, if that gets you real compromises, the protocol is considered broken.
End to end encryption protects content. It doesn't protect things like information about who is talking to who. For that you need something like an onion network. As already mentioned, Briar uses Tor for that. Signal claims to not collect such information but my point is that we have no way to know what they collect. Claims don't count for anything for these sorts of things..
