The difference is that you can be reasonably sure that an email will be delivered to the user's address that they specified, and there is a common understanding that sharing your email credentials will result in immediate financial loss. The communication is out-of-band, as SMTP is not an authentication protocol, so the user must perform some affirmative action (such as clicking a link in the email) to authenticate the request.
With SSO, the claim that a user is authentic lies completely with the provider. If there is a bug in the implementation, a user could be authenticated when they should not be, and the actual account owner may never even be notified that such an authentication occurred.
With oauth/oidc, the service that's requesting the token doesn't even specify the username. What would the IdP send back if the user is not authenticated? The concern is it will just issue a token for a random user? Or that someone will break into another account on the IdP's login form?
I don't see any difference with email besides convenience. If an email provider has a bug, they might allow someone else to read your email and click the link without any notification to the actual account owner that such an authentication occurred. In fact, if a provider has oauth and webmail, the two probably use the same authentication system (e.g. they use the same session cookie and login form, or the webmail might use the oauth service as its auth). Even native email clients are using oauth these days. This is a good thing. It lets you store a token with limited permission in your email client instead of storing a password for SMTP that also gives you full permissions to everything the account can do.
Also most oauth implementations I've seen actually do have a dashboard in your user settings where they record which services you've provided tokens to, and what scopes those tokens have.
If you're worried about CSRF, that's what the state parameter is for, and it's the consuming service that needs to use it appropriately, so you don't need to worry about whether the IdP did it right. If the IdP gives the wrong state back, the login will fail when your service checks it.
