To answer my own question: Yes, WebRTC supports peer-to-peer video calls but does require a “signaling server” to help establish and close the peer-to-peer connection. [1]
The signaling server is only useful for the connection initialization though, and it never has access to the video stream, only to metadata (like your IP address, the supported encodings of each party, etc.). And it doesn't even really need to access them: it just needs to forward them from one peer to another, so it could be end to end encrypted.
It also needs one or several STUN servers as part of the hole punching scheme, but this one doesn't even exchange anything with anyone, so there aren't many issues here (and you don't need to roll your own: you can use Google's one)
[1]: https://www.html5rocks.com/tutorials/webrtc/basics/ "Getting Started With WebRTC"