The AI answers are nowhere near good enough to always be at the top, without any clear indication that they are just a rough guess. Especially for critical things like visa requirements or medical information. When you search Google for these sort of things, you want the link to the authoritative source, not a best guess. It’s very different for queries like say “movies like blade runner”.
I doubt that was the decision process. It’s much more likely that there is a directive coming down from the top that “we need to go all in on AI”, which then gets amplified down by middle management and the result is AI smeared over all results irregardless if helpful. That then drives up some vanity metric like “searches answered by AI summaries”, while metrics like “bad AI summaries shown” don’t get attention. As a result the organization is happy, people can get promoted, etc.
Not all queries are the same but I agree with you that the authority of source is crucial. That's why for example .gov sites rank high and should rank high because government is usually the most trusted source.
But when you are looking for new shoes to buy or food recipes then .gov sites can't help you and that's where things get ugly....SEO spam ugly.
An example: I was looking up what a good diet is to follow after a child has been vomiting. The AI said to avoid giving fruit juice … yet the authoritative sources said the opposite. I already knew not to trust the AI, but this was nail in the coffin for me.