This is being worked on. AFAIK, Apple is the first to incorporate this approach into a released product, with the Screen Recognition feature of VoiceOver starting in iOS 14.
Thanks for the response Matt. I leave the link here for others to look into [1].
Their effort seems currently limited to iOS based Phone screens. iOS is perhaps easier to solve given the strong Apple design guidelines for apps to pass the App Store review process.
Perhaps a community supported distributed approach to help build the database of annotated screens for the model to learn from, combined with open source models for all kinds of screens and applications( not just Apple) would be interesting project to work on.
Interestingly, iOS screen recognition also allows exploration of screenshots and remote desktop screens. I've heard of people using it to remotely install Windows as well. It would surprise me if Apple didn't have plans to put the same feature into its M1 mac.
