Usually you can just import the leaf self signed cert as a CA in your OS trust store and the problem goes away (assuming it has an IP SAN). Slightly tedious but you can issue the certs with long validity

Chrome provides no simple way to trust a self-signed cert. When you go to certificate details, the only option under the "Details" tab is "Export...". The only work around is to click "Advanced" and "Proceed to example.com (unsafe)". Chrome will then helpfully suffer amnesia in 1-3 days and completely forget you want to allow an exception for the certificate.

iirc Chrome uses OS trust store so the trust needs to be using the operating system's facility

That sounds like a defect in the browser design.

Or maybe it's because you actually want an identity to verify (which an IP address is not.)