What would this list contain? You don't have a signal username. If you did you'd have to claim it somehow (degenerates to email or phone verification). It's not that simple.

Using phone numbers allows signal to plug into the existing state of the world and leverage it to upgrade the security of messaging for everyone who uses it. The one compromise is that it treats phone number as a short identifier (importantly, not cryptographic, it uses real crypto for that).

If you don't use phone numbers, your product would look more like Keybase. You have to somehow facilitate key exchange between people in a way that's actually usable. Otherwise all your security benefits go out the window because nobody uses your product. Signal understands this nuance perfectly which is why they're a successful product.

If I don't use Signal, then I'm not going to keep a list of my friends' Signal usernames on my phone.

If I subsequently sign-up for Signal, then I have no way to discover which of them use Signal - short of contacting them via some other method and asking for their Signal username, if any.

By making the Signal username the same as the user's phone number, I actually DO have a list of Signal 'usernames' on my phone already. As soon as I sign-up, I can send my list of friends' phone numbers to Signal and they can tell me which of those people have Signal accounts.