I think this is pretty cool. Somewhat similar tools have been used to cheat in MMOs where a library would recognize and interpret the pixels and you'd write scripts to manipulate the underlying platform.

This is actually extremely flexible and could be used not just to decorate a user interface, but to provide a programmatic /scripted/ interface into an application that doesn't traditionally provide an API.