That would be fine if browsers didn't throw up giant warning signs when using self-signed certificates.

Usually you can just import the leaf self signed cert as a CA in your OS trust store and the problem goes away (assuming it has an IP SAN). Slightly tedious but you can issue the certs with long validity

Chrome provides no simple way to trust a self-signed cert. When you go to certificate details, the only option under the "Details" tab is "Export...". The only work around is to click "Advanced" and "Proceed to example.com (unsafe)". Chrome will then helpfully suffer amnesia in 1-3 days and completely forget you want to allow an exception for the certificate.

iirc Chrome uses OS trust store so the trust needs to be using the operating system's facility

That sounds like a defect in the browser design.

Or maybe it's because you actually want an identity to verify (which an IP address is not.)

And this protects you from a hostile network how?

How does the certificate? If you already have to do the TLS handshake it doesn't change anything.

A verified certificate lets you know you didn't handshake with an attacker in the middle.

Let me rephrase that: How is the CA supposed to know they didn't handshake with an attacker? All they have is the IP, there's no identity to check like with DNS.

The CA connects to the IP from multiple different points across the internet. If you can convince all of them, you almost certainly do control the IP.

You as a normal client don't do that. Your computer can be fooled by very easy local spoofs.

And for what it's worth, taking over the IP would also let you get a DNS-based certificate, so those actually have more weak points.