"Google regularly fails to include all words I searched (even if it's only three or four), often retrieving completely useless results. I doubt it's due to incompetence; I take it as a signal that they're now struggling to match the volume and characteristics of the data they have to ingest to an adequate user experience."

That is awfully generous of you ...

Google shows us what it shows us to maximize advertising revenue. They need you to keep clicking and generate hits on adwords-encumbered websites. Showing you zero (or one or a handful of) results for your search query is counter to this goal.

Showing you a batch of results with no adwords-encumbered sites in it is also counter to this goal.

I don't think they're struggling with anything at all - they are optimizing for paid clicks and precise search results is, at best, a very distant second priority...