I believe what you're looking for for SSO is RFC 7591[0], which describes how to register with an oauth IdP on the fly. RFC 8414[1] describes well-known locations to get metadata about the registration process. So the standards are there, and in theory you could have a login form where e.g. someone types their email (or the browser autofills it), and that kicks off an oauth login to that domain, doing client registration on the fly if the server has never talked to that domain before. I've never seen it in the wild though. Would be nice.
I think tailscale supports exactly that? You give them your email address and they check the web finger endpoint of the domain for your configured SSO provider. https://tailscale.com/kb/1240/sso-custom-oidc
[0] https://datatracker.ietf.org/doc/html/rfc7591
[1] https://datatracker.ietf.org/doc/html/rfc8414