Sure, that's true, and I bet a Down-Syndrome-recognizing neural net could be made quite accurate. I guess what I mean to say is that there's no general mechanism that would cause someone's inner character to be reflected in their appearance in any consistent way. If you want to predict behavior from appearance, you have to actually do the science and prove that there's an underlying mechanism before you can trust that your neural net is recognizing something meaningful.