The assumption is that Alice recognizes the voice of Bob. If Eve manages to evasdrop on the call and sits in the middle or beats Bob to connect to the wormhole server then Alice will still see that the fingerprint that Bob dictates over the phone does not match the fingerprint of the key that her computer proposes to use for the file transfer. Alice will therefore abort the transmission.

With deep learning the voice may be not good enough nowadays. Still, you only need an authenticated - possibly public - channel, similar to pgp key exchange, where you can read the fingerprint over the phone.