My lab is trying to do something similar for answering questions about images. We have a significantly better system than the current system that's online, but we haven't had a chance to update it yet: http://askimage.org

It is far from perfect, but is near state-of-the-art. I'm guessing it won't hold up to HN.